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Advanced  Physiological  Estimation  of  Cognitive  Status  (APECS):  Final  Report 

Feonard  J.  Trejo,  Roman  Rosipal,  and  Paul  F.  Nunez 

Pacific  Development  and  Technology,  FFC 
999  Commercial  Street,  Suite  205,  Palo  Alto,  Ca  94303 
ltrejo@pacdel.com 

1  Introduction 

Impairment  of  cognitive  performance  in  individual  warfighters  during  a  mission  poses  a  high  risk  for 
procedural  errors,  for  which  the  Army  currently  lacks  real-time  countermeasures.  Such  cognitive 
impairment  has  been  documented  in  previous  missions,  and  attributed  to  several  factors,  including 
time  on  duty,  sleep  loss,  extended  time  on  single  tasks,  unusually  high  workload,  psychosocial  or 
combat  stress,  exposure  to  neurotoxins  and  vestibular  dysfunction.  To  address  this  problem,  the 
Army,  DARPA,  and  other  DOD  agencies  have  been  developing,  testing  and  refining  systems  that 
automatically  monitor  the  physiological  and  cognitive  status  of  individual  soldiers. 1,2,3  These  systems 
use  physiological  sensors  to  track  soldiers’  health  and  mental  fitness  and  transmit  data  to  command 
and  control  systems. 

Some  of  the  signals  that  the  physiological  sensors  measure  are  readily  interpreted,  such  as  estimation 
of  the  heart  rate  or  heart-rate  variability  from  the  electrocardiogram  (ECG).  However,  the  estimation 
of  cognitive  workload  from  the  electroencephalogram  (EEG)  and  other  signals  requires  a  complex 
series  of  mathematical  transformations  or  algorithms.  Overall,  research  on  algorithms  for  estimation 
of  cognitive  status  has  made  remarkable  progress.  For  example,  certain  kernel  partial  least  squares 
(KPFS)  algorithms  classified  physiological  states  corresponding  to  states  of  mental  engagement  or 
cognitive  fatigue  using  a  few  seconds  of  EEG  and  other  signals  with  90-100%  accuracy  and  good 
test-retest  reliability.4,5,6  However,  algorithms  to  estimate  cognitive  workload  are  typically  only  about 
60-70%  accurate  under  realistic  test  conditions.7,8 

For  example,  Pacific  Development  and  Technology  (PDT)  recently  applied  a  KPLS  algorithm  to 
estimate  engagement  and  workload  for  the  US  Army  Toxins  II  SBIR  program.1  We  found  that  the 
EEG  features  that  previously  worked  for  classification  of  engagement  or  fatigue  failed  to  classify 
workload.8  However,  due  to  limitations  in  the  scope  of  that  program,  we  could  not  study  several 
promising  avenues  for  adaptation  of  the  algorithm  to  workload.  In  particular,  we  identified  two 
innovations  that  have  the  potential  for  isolating  and  measuring  neurophysiological  mechanisms 
important  for  cognitive  workload.  The  first  innovation  is  to  apply  a  biophysical  model  of  EEG 
coherence  among  neuronal  networks  that  participate  in  cognitive  processes.9  The  second  innovation 
is  to  apply  new  multi-way  analyses  that  more  accurately  model  the  interaction  of  interdependent 
processes  than  previous  two-way  methods. 10,11 

2  Objective 

This  project  addressed  two  requirements  described  in  ARO  BAA  W91 1NF-07-R-0003: 

3.7.4.  Human  Cognitive  and  Behavioral  Modeling.  Quantitative,  analytical  models  of 
cognition  and  behavior  are  required  for  training,  simulation  ...  and  mission  planning. 

8A  Neurophysiology  and_  Cognitive  Neuroscience.  Research  in  the  perception  and  cognition 
subfields  of  neurophysiology  and  the  cognitive  neurosciences,  ...  physiological,  ...  and/or 
cortical/cognitive  mechanisms  underlying  successful  completion  of  complex  task  behaviors 
applicable  to  non-laboratory  environments  under  non-ideal  conditions,  to  include  both 
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amelioration  of  induced  losses  as  well  as  enhancement  in  defined  perceptual,  cognitive  and/or 
motor  abilities. 

In  service  of  these  requirements  we  developed  new  algorithms  for  Advanced  Physiological 
Estimation  of  Cognitive  Status  (APECS),  which  significantly  improved  the  estimation  of  cognitive 
workload  and  shed  new  light  on  the  estimation  of  mental  fatigue.  More  specifically,  we  used  atomic 
decomposition  to  identify  unique  sources  of  brain  electrical  activity  as  measured  by  the  EEG 
recorded  in  human  participants  as  they  performed  tasks  that  induced  different  mental  states, 
including  engagement,  mental  workload,  and  mental  fatigue.  We  tested  two  types  of  atomic 
decomposition,  each  of  which  identifies  unique  EEG  sources  simultaneously  in  three  dimensions:  1) 
atoms  with  dimensions  of  power  spectral  density,  space  (electrode  position),  and  time  (time  on  task 
or  task  conditions),  or  2)  atoms  with  dimensions  of  magnitude  squared  coherence,  spatial 
relationships  (electrode  pairs),  and  time.  For  tasks  that  induced  mental  workload,  we  found  atoms 
that  combine  sources  in  the  theta  (4-8  Hz)  and  alpha  (8-12  Hz)  EEG  frequency  bands  consistently  in 
individual  participants  at  different  times  of  day  and  on  different  days.  The  temporal  variations  of  the 
atoms  clearly  reflected  the  levels  of  mental  workload  induced  by  varying  task  conditions.  For  a  task 
that  induced  mental  fatigue,  we  found  atoms  that  tracked  the  development  of  mental  fatigue  in 
individual  participants  over  time,  while  reflecting  underlying  changes  in  power  or  coherence  of 
primarily  theta-band  EEG. 

Our  results  show  that  atomic  decomposition  is  a  valuable  new  approach  to  the  identification  and 
measurement  of  EEG  sources  for  monitoring  cognitive  status.  By  comparing  these  results  with 
results  of  prior  analyses  using  the  same  data  sets,  we  observed  that  atomic  decomposition  can 
supplement  or  overcome  existing  approaches  based  on  conventional  two-dimensional  space-time  or 
frequency-time  decomposition  of  EEG. 

We  foresee  the  future  development  of  APECS  algorithms  for  occupational  use  or  performance 
optimization  as  comprising  several  stages.  Initially,  we  must  evaluate  the  APECS  algorithms  in  a 
wider  range  of  experimental  contexts,  and  over  a  much  wider  range  of  individual  participants.  In 
parallel,  APECS  algorithms  may  be  embedded  in  wearable  systems  that  use  advanced  sensors  and 
intelligent  software  to  gather  preliminary  field  estimates  of  cognitive  status  of  individual  warfighters 
in  operational  settings.  The  interplay  of  these  parallel  efforts  -  one  laboratory  based  and  the  other 
field  based  -  will  allow  us  to  “harden”  the  algorithms,  both  in  terms  of  scientific  accuracy  and 
operational  utility.  Ultimately,  through  the  use  of  interactive  simulations,  such  as  immersive  virtual 
reality,  the  APECS  algorithms  may  be  adapted  to  provide  continuous  feedback  of  cognitive  status 
that  can  serve  to  enhance  or  sustain  cognitive  performance.  Prior  research  has  shown  that  such 
feedback  may  enhance  situational  awareness,  sustain  cognitive  engagement  in  supervisory  control 
tasks  and  lower  the  risk  of  errors  associated  with  fatigue,  inattention,  or  overload.  12,13,14 

The  APECS  algorithms  are  presently  specified  in  high-level  MATLAB  code  that  can  readily  be 
cross-compiled  to  run  on  different  hardware  systems.  Thus,  ultimately  the  APECS  algorithms  may 
serve  as  “plug-ins”  for  biosensor  or  feedback  systems  from  different  vendors  and  serve  to  ensure  and 
enhance  warfighter  performance  as  part  of  the  Army’s  future  command  and  control  systems. 

3  Background 

Recently,  prototype  systems  for  real-time  estimation  of  cognitive  states  have  been  developed  by 
academic,  industrial,  and  government  participants  in  the  DARPA  Augmented  Cognition  Program,15 
NASA,  and  other,  focused  DOD  programs.  These  systems  can  use  changes  in  human  behavioral  and 
physiological  patterns  to  detect  hazardous  states  such  as  cognitive  fatigue,  disengagement 
(inattention),  or  cognitive  overload.  Recent  NASA  studies  have  shown  that  with  as  little  as  3.5 
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seconds  of  EEG  data,  a  robust  multivariate  algorithm  correctly  identified  90  to  100%  of  periods 
during  which  individuals  performing  a  demanding  3-hour  task  experienced  cognitive  fatigue.5,6  A 
PDT  variant  of  this  algorithm  accurately  classified  fatigue  in  Air  Force  pilots  over  a  37-hr  vigil  and 
proved  to  be  resistant  to  noise  and  reduced  sensor  density.16  Other  NASA  research  had  shown  that 
real-time  feedback  derived  from  physiological  measures  of  mental  engagement  prolonged  or 
enhanced  human  performance  during  supervisory  control  of  automated  systems. 12,13,14  Currently, 
scientists  at  the  Institute  for  Human  and  Machine  Cognition  (IHMC),  Honeywell,  and  elsewhere  are 
integrating  physiological  real-time  “cognitive  gauges”  with  C3I  tasks.3,17,18  As  PI,  co-I  or  advisor  for 
projects  in  all  of  these  programs,  Dr.  Trejo  personally  directed  the  development  and  testing  of 
algorithms  for  estimation  of  cognitive  status,  and  contributed  to  significant  technology  advancements. 
His  direct  experience  in  these  programs  also  allows  him  to  frankly  assess  the  current  limitations  and 
the  promise  of  new  research  directions. 

The  ARO  now  has  a  timely  opportunity  to  build  on  existing  results  and  take  a  significant  step 
forward  in  the  estimation  of  cognitive  workload.  Four  factors  contribute  to  this  opportunity:  1) 
progress  in  robust  algorithm  development,  2)  advances  in  the  theory  of  EEG  coherence  and  its 
relationship  to  neurocognitive  processes,  3)  advances  in  multi-way  methods  for  neuroimaging,  and  4) 
the  availability  of  multiple  databases  for  algorithm  development  and  validation. 

RobustAlgorithm  Development.  First,  at  PDT  we  have  made  substantial  progress  in  formalizing  a 
process  of  robust  algorithm  development.  We  now  have  a  library  of  unique  high-dimensional  feature 
extraction  methods  and  robust  linear  and  nonlinear  classifiers  specifically  for  processing 
neurocognitive  data.  Our  design  for  the  APECS  algorithms  builds  on  12  years  of  cutting-edge 
research  on  modeling,  signal  processing,  and  machine  learning  methods  for  human  physiology  and 
cognition.  Drs.  Rosipal  and  Trejo  published  the  seminal  article  in  the  field  of  KPLS  regression,  and 
extended  this  to  cognitive  task  analysis,19  estimation  of  cognitive  fatigue, 6  and  brain-computer 
interfaces.20  KPLS  finds  robust  mappings  between  high-dimensional  inputs,  such  as  multimodal 
physiology,  and  low-dimensional  outputs,  such  as  cognitive  states.  KPLS  also  uses  non-linear 
“kernels,”  to  transform  high-dimensional  features  and  allow  for  mapping  of  nonlinear  relationships. 
When  coupled  with  classifiers,  such  as  Fisher’s  discriminant  or  support  vector  classifiers,  KPLS 
yields  robust  classification,  with  fast  computation  as  needed  for  real-time  learning  or  processing.21 

Shortz  and  Long-Range  EEG  Coherence.  The  theory  of  short-  and  long-range  coherence  is  an 
important  development  in  modeling  the  macroscopic  properties  of  brain  function  that  reflect 
cognitive  processes,9  which  has  yet  to  be  rigorously  applied  to  cognitive  workload  estimation. 
However,  some  simple  experiments  have  shown  that  the  frequency  bandwidth,  spatial  range,  and 
spatial  direction  of  EEG  coherence  functions  can  discriminate  different  cognitive  states.22  Specific 
processes  that  are  associated  with  coherence  include:  selective  attention,  transfer  of  information 
among  different  but  coherent  networks,  binding  of  neuronal  groups  for  unitary  processing,  such  as 
object  perception  or  working  memory,  and  plasticity  subserving  learning  and  memory. 

As  described  in  the  Results  (Section  05),  we  have  confirmed  that  the  inclusion  of  short  and  long- 
range  EEG  coherence  features  in  algorithms  for  estimating  cognitive  workload  improves  on 
algorithms  that  rely  on  the  EEG  power  spectrum  alone.  One  of  the  problems  that  have  plagued 
cognitive  workload  estimation  is  that  changes  in  EEG  band  power  can  mean  different  things  in 
different  contexts,  leading  to  inconsistency  of  the  feature  weights  used  to  classify  EEG  segments.  For 
example  it  is  widely  known  that  alpha  band  power  decreases  with  engagement  and  cognitive 
workload,23  but  in  other  cases  increases  in  alpha  power  and  coherence  may  reflect  specific  memory 
processes.24  Without  due  consideration  of  the  scale  and  direction  of  coherence  changes,  EEG 
measurements  can  have  an  inconsistent  connection  to  cognition.  However,  the  conflicting  results  of 
these  studies  are  reconciled  by  a  short/long-range  coherence  model  in  which  local  oscillatory 
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dynamics  interact  with  global  modulations  of  synaptic  action  in  multiple  frequency  bands  to  produce 
observed  EEG.22 

Multi-way  Methods.  Another  important  development  is  the  successful  application  of  multi-way 
methods  such  as  N-way  Partial  Least  Squares  (N-PLS)  or  parallel  factor  analysis  (PARAFAC)  to 
brain  activity  such  as  EEG  or  fMRI. 10,25  N-PLS  is  multi-way  extension  of  PLS  which  combines  the 
class-specific  feature  extraction  power  of  PLS  while  preserving  the  multi-way  structure  of  the 
variance  sources  (supervised  learning).  PARAFAC  is  a  multi-way  extension  of  factor  analysis  that 
extracts  features  without  respect  to  class  criteria  (unsupervised  learning).  In  theory,  multi-way 
methods  can  disambiguate  neurocognitive  processes  that  may  be  confounded  by  two-way  methods 
such  as  principal  components  analysis  (PCA),  independent  components  analysis  (ICA),  or  ordinary 
PLS.  For  example,  the  standard  approach  to  EEG  analysis  for  estimation  of  cognitive  state  is  to 
analyze  the  frequency  spectrum  and  location,  or  frequencyxchannel  where  a  channel  is  an  electrode 
or  electrode  pair.  When  a  third  dimension  such  as  time  on  task  (or  conditions)  is  added,  the  formal 
model  becomes  a  matrix  of frequency  x  channel  x  time.  Two-way  methods  can  only  decompose  the 
three-way  interaction  by  collapsing  frequency  and  channel  into  one  dimension,  resulting  in  a  two- 
way  matrix  of  ( frequency-channel )  x  time.  The  result  is  a  series  of  frequency-channel  components 
mixed  in  different  proportions  for  each  sample  time.  Such  mixing  of  dimensions  obscures  the  correct 
interpretation  of  a  component,  or  even  worse,  can  fail  to  identify  phenomena  that  are  specific  to  one 
of  the  arbitrarily  mixed  dimensions.  Furthermore,  components  so  obtained  are  not  uniquely 
determined,  and  must  be  constrained  artificially  by  criteria  such  as  orthogonality,  which  are  not 
likely  to  fit  neurocognitive  processes.  Multi-way  methods,  on  the  other  hand,  yield  a  unique 
decomposition  that  preserves  the  uniqueness  of  each  dimension  of  the  measurement  matrix.26  In  this 
way  multi-way  methods  represent  EEG  sources  corresponding  to  a  neurocognitive  process  with  an 
atomic  structure,  where  each  atom  represents  a  unique  source  of  variance  in  the  multi-way  matrix. 
The  N-PLS  and  PARAFAC  methods  have  been  used  to  identify  EEG  alpha-,  theta-,  and  gamma-band 
“atoms”  that  coincide  with  simultaneous  fMRI  estimates  of  spatial  location,  with  similar  degrees  of 
efficacy.27 

Neuwcognitive  Databases .  We  now  have  access  to  five  databases  from  controlled  studies  of 
physiology  and  human  performance,  including  four  studies  that  focused  on  estimation  of  cognitive 
workload  and  detection  of  cognitive  overload.  Drs.  Trejo  and  Rosipal  directed  experiments,  wrote 
algorithms  or  analyzed  data  from  each  of  these  studies  and  are  uniquely  familiar  with  the  databases. 
We  used  three  of  these  databases  to  develop  and  test  APECS  algorithms.  All  data  from  these  studies 
were  collected  with  informed  consent  and  approved  IRB  protocols.  The  data  were  coded  such  that 
the  identities  of  the  participants  will  be  unknown  to  PDT  analysts.  The  three  databases  used  in  the 
present  project  included: 

•  USAF-C:  C2ISR  Multimodal  Study  of  UAV  Operator  Readiness  (6  multi-test  participants17) 

•  USA-T:  Army  Toxins  II  Multimodal  Study  of  Cognitive  Overload  (8  test-retest  participants4) 

•  NASA-C:  NASA  Cognitive  Fatigue  Database  (16  participants,  public  domain6) 

Two  additional  databases,  which  manipulated  cognitive  workload,  may  be  used  in  future  research: 

•  USN-B:  Navy  Biopsychometric  Assessment  Program  Database  (8  test-retest  participants28) 

•  NASA-E:  NASA  ERTAS  Database  (8  test-retest  participants13) 
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4.1  Task  1.  Analysis  of  Tasks,  Workload  Metrics,  and  Physiological  Data 

The  aim  of  this  task  was  to  decompose  each  experimental  database  to  define  workload,  time 
resolution,  and  physiological  metrics.  For  each  database,  we  followed  a  three-step  process  of 
algorithm  adaptation  and  testing  (Figure  1).  First,  we  analyzed  each  task  from  a  neurocognitive 
standpoint  to  guide  our  selection  of  workload  and  physiological  metrics  and  time  resolution.  For 
example,  the  USAF-C  database  includes  visual  signal-detection,  working  memory,  and  executive 
control  tasks.  Perceptual  tasks,  such  as  signal  detection,  would  activate  visual-spatial  processing 
networks  located  in  occipital,  parietal,  and  inferotemporal  cortex,  which  will  modulate  alpha  1  and 
alpha  2  powers  and  coherence.  The  working  memory  demands  of  the  USAF-C  task  would  activate 
the  anterior  cingulate  and  dorsolateral  prefrontal  cortex,  producing  increased  midline  frontal  theta 
rhythms. 24,29  This  prior  knowledge  of  functional  specialization  guided  our  selection  of  EEG 
electrode  sites,  frequency  bands,  and  spatial  /  temporal  resolution  for  power  and  coherence  estimates. 

The  temporal  resolution  of  EEG  analysis  for  each  task  was  long  enough  to  reliably  estimate 
parameters  but  short  enough  to  detect  changes  related  to  task  conditions.  By  applying  these  principles 
to  each  task,  we  produced  a  set  of  analysis  parameters  (Table  1). 


Task 

Code 

Perceptual  &  Cognitive 
Processing  Demands 

Neural 

Sources 

EEG 

Bandwidth 

Range  Of 
Temporal 
Resolution 

Minimum 

Frequency 

Resolution 

USAF-C 

Visual  signal  detection 

Parietal- 

occipital 

8-25  Hz 

2-3.5  s 

1.0  Hz 

Working  memory  & 
executive  control 

Fronto 

central 

4-8  Hz 

8-12  Hz 

2-3. 5s 

2-8  s 

1.0  Hz 

0.5  Hz 

USA-T 

Visual  signal  detection 

Parietal- 

occipital 

8-20  Hz 

2-3.5  s 

1.0  Hz 

Auditory  language 
processing 

Temporal- 

central 

8-20  Hz 

2-3. 5s 

1.0  Hz 

Working  memory  & 
executive  control 

Fronto 

central 

4-12  Hz 

2-8s 

1.0  Hz 

NASA-C 

Working  memory  & 
executive  control 

Fronto 

central 

4-18  Hz 

2-13  s 

0.5  Hz 

Table  1 .  Spatial,  frequency,  and  temporal  analysis  parameters  for  the  three  task  databases. 


4.2  Task  2.  APECS-W  Algorithm  Adaptation 

As  suggested  by  a  current  theory  of  local/global  EEG  coherence9  and  experiments  on  EEG  and 
cognitive  function,22  we  hypothesize  that  cognitive  workload  is  reflected  by  the  desynchronization  of 
a  parietal  alpha  atom  defined  by  long-range  coherence  with  frontal  regions  and  the  synchronization 
of  a  frontal  midline  theta  atom  defined  by  local  coherence  with  neighboring  frontal  regions.  Using 
this  guiding  hypothesis  and  the  analyses  from  Task  1,  we  structured  two  APECS-W  algorithms  for 
each  database.  In  the  first  algorithm,  or  APECS-Wp,  aimed  at  power  spectral  density  effects,  the 
three-way  input  matrix  consisted  of  EEG  power  spectral  densities  for  frequency  bins  from  1-25  Hz, 
electrode  position,  and  time  of  measurement  (which  reflected  task-induced  workload  transitions).  In 
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the  second  algorithm,  or  APECS-Wc,  aimed  at  spectral  coherence  effects,  the  three-way  input  matrix 
consisted  of  multi-scale  EEG  coherence  spectra,  electrode-pair  (all  unique  pairs  of  electrodes 
excluding  self-pairing),  and  time.  For  the  NASA-C  database  we  structure  two  similar  algorithms, 
APECS-Fp  and  APECS-Fc,  using  similar  principles,  but  different  constraints  (see  below). 


1  Analy7P  DrtahsK#*  ft  Dpfinp Metric*; 


♦  N  euro  cognitive 
Anaysis 

♦  Workload  Metrics 

♦  Physiologica  Metrics 

♦  Terrporal  Resolution 


J 


7.  Algorithm  Adaptation 


•  Preprocessing 

•  Feature  Selection 

•  Feature  Extraction 

•  Normalization 

•  Structure  Order 


J 


3.  Algorithm  Training  &  Testing 

♦  DataPartitions 

•  Cross-validation 

♦  Train  &  Test 
•TcstRobustrcss 

♦  Noise 

•  sensorLoss 

•  Stabilization 

\ - J 


Figure  1.  Three-step  process  of  APECS-W  algorithm  adaptation  and  testing.  In  Step  1,  we  use 
neurocognitive  theory  and  data  to  analyze  tasks  to  define  workload,  time,  and  physiological  metrics.  In 
Step  2,  we  will  define  preprocessing,  feature  selection/extraction,  normalization  methods  and  adapt  the 
APECS  structure  and  order  to  the  features.  In  Step  3  we  will  select  random  training,  cross-validation,  and 
testing  partitions  then  train  and  test  the  algorithm.  In  future  studies  we  may  also  test  the  resulting 
algorithm  for  tolerance  of  noise  and  sensor  loss,  and  then  apply  optional  stabilization  methods  to  improve 
test-retest  reliability. 

In  the  present  analyses  the  outputs  consisted  of  unique  atoms  for  which  we  obtained  the  loadings 
using  PARAFAC  decomposition  with  the  following  constraints: 

•  We  corrected  EEG  records  for  ocular  artifacts  and  segmented  them  into  non-overlapping 
contiguous  epochs  of  2-s  duration,  providing  frequency  resolution  of  0.5  Hz.  This  resolution  was 
satisfactory  for  all  task  requirements  (Table  1). 

•  We  performed  outlier  detection  by  removing  points  with  unduly  high  leverage.  For  this  we  did  an 
initial  PARAFAC  decomposition  and  then  clipped  data  points  for  which  individual  loadings  in 
any  one  dimension  exceeded  a  fixed  percentile  of  the  populations  of  loadings  for  that  dimension 
in  the  given  experiment.  The  percentiles  we  used  ranged  from  90  to  99%.  After  removing  such 
points  we  did  a  second  decomposition  and  retained  the  loadings  as  results. 

•  We  imposed  a  constraint  of  non-negativity  for  loadings  on  all  dimensions  of  the  PARAFAC 
decompositions.  In  addition,  for  the  NASA-C  dataset,  coherence  analysis,  we  used  an  additional 
constraint  of  unimodality  on  the  frequency  dimension. 

•  We  did  not  rescale  or  normalize  data  for  any  dimension.  However,  for  display  purposes  only,  we 
scaled  loadings  for  the  time  dimension  for  comparing  atoms  with  very  different  loading  means 
and  variances  within  experiments. 

•  Units  of  power  spectral  density  were  dB/ a/  Hz  .  Units  of  coherence  were  the  conventional 
dimensionless  units  of  magnitude  squared  coherence  ranging  in  value  from  0  to  1,  where  0  means 
no  coherence  and  1  means  perfect  coherence  at  a  given  frequency. 

Two  approaches  were  used  to  assign  initial  values  to  the  loadings  before  iterating  solutions  to  the 
PARAFAC  decompositions.  For  both  APECS-W  algorithms  we  used  a  method  of  performing  several 
small  runs  then  averaging  the  resulting  loadings  and  using  the  averages  to  initialize  the  loadings  for 
complete  decompositions.  For  the  APECS-F  algorithm,  we  used  singular  value  decompositions  in 
each  dimension  to  estimate  the  initial  loadings. 
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For  APECS-W  we  used  a  uniform  convergence  criterion  of  1.0X10'6  for  iterating  the  algorithm,  i.e., 
iteration  ceased  when  the  change  in  total  variance  explained  was  less  than  1/1 0,000th  of  one  percent. 
For  APECS-F  the  convergence  criterion  was  0.001  or  an  improvement  in  the  model  fit  of  less  than 
1/1  oth  of  one  percent. 

After  preprocessing  all  EEG  records  to  remove  EOG  artifacts  we  performed  three-way  unsupervised 
PARAFAC  decompositions  to  identify  the  atoms  (multidimensional  components  of  variance)  in  the 
three-way  input  matrices.  For  the  APECS-Wp  and  APECS-Fp  algorithms,  the  EEG  atoms,  A,  are 
defined  as  three-way  sources  with  dimensions  of  frequency,/  electrode,  e,  and  time,  t  (Eq.  1).  Each 
atom  is  estimated  by  two  normalized  vectors  (a,  b ),  a  score  vector  c  and  a  noise  term,  £eft. 


Nk 

A  eft  —  '^JaekbfkCtk  +  ^  eft  (Eq.  1) 

k= 1 


4.3  Task  3.  Algorithm  Training  and  Testing 

The  structure  of  the  APECS-W  algorithm  was  similar  for  each  database,  but  we  adapted  the  variable 
inputs,  outputs,  and  preprocessing  requirements  of  each  database  to  a  format  that  was  suited  for 
cross-study  validation.  Since  all  of  the  decompositions  we  did  were  unsupervised,  there  was  no  need 
for  formal  training,  testing,  and  validation  sets.  However,  for  the  USAF-C  and  USA-T  datasets  we 
used  sessions  performed  at  different  times  of  day  and  on  different  days  for  this  purpose.  For  example, 
we  used  session  data  from  Day  1  to  estimate  the  atoms  and  then  projected  the  EEG  data  for  Day  2 
using  the  weight  vectors  for  each  atom  to  reproduce  the  time  course  of  each  atom  across  workload 
conditions.  No  test-retest  validations  were  performed  as  of  yet  for  the  NASA-C  data  set. 

5  Results 

5.1  Synopsis  of  Results 

In  this  project  we  designed  and  tested  an  APECS  workload  algorithm  (APECS-W),  to  increase  the 
accuracy  and  reliability  of  estimated  cognitive  workload  and  detect  periods  of  cognitive  overload. 

We  aimed  for  a  20%  increase  in  accuracy  and  test-retest  reliability.  Our  minimally  successful 
criterion  was  10%.  Our  prior  simulations  showed  that  even  a  10%  improvement  will  move  estimation 
of  workload  near  the  lower  range  of  accuracy  now  possible  for  estimation  of  engagement  or  fatigue 
(Figure  2).  We  considered  distinguishing  engagement  from  workload,  but  our  focus  was  on 
discriminating  workload  states  pertaining  to  active  task  engagements.  We  tested  the  APECS-W 
algorithm  using  two  databases  (USAF-C,  USA-T)  described  above.  A  formal  classification  analysis 
was  outside  the  scope  of  this  STIR  project,  so  we  made  informal  estimates  of  accuracy  based  on 
inspection  of  the  data  and  comparisons  with  prior  results.  We  will  perform  formal  classification 
analyses  in  our  future  development  of  the  APECS  algorithms.  For  now,  our  experience  with  all  of  the 
data  sets  examined  here  suggests  that  there  was  an  improvement  of  more  than  20%  in  the  estimation 
of  cognitive  workload  using  atomic  decomposition,  as  compared  to  our  prior  methods  using  two-way 
analyses.  For  the  fatigue  data,  we  estimated  a  lower  level  of  improvement,  which  is  to  be  expected 
from  the  already-high  accuracies  of  prior  classifications. 

We  designed  the  APECS  algorithm  to  be  adaptable  to  a  wide  range  of  tasks  that  require  human 
performance  or  supervision.  In  particular,  we  adapted  the  algorithm  to  the  estimation  of  mental 
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fatigue,  and  will  refer  to  this  algorithm  variant  as  APECS-F.  Although  we  did  not  set  out  to  compare 
APECS-F  quantitatively  with  prior  results,  we  report  below  that  APECS-F  was  highly  successful  in 
identifying  EEG  atoms  that  track  the  development  of  mental  fatigue  in  individual  participants. 

5.2  Detailed  Results 

5.2.1  USAF-C  Database  -  Initial  Published  Results 

We  presented  results  of  our  initial  analyses  using  two  participants  from  the  USAF-C  database  at  the 
recent  PLS09  conference  and  published  a  corresponding  full-length  paper  in  the  edited  book  of  the 
conference  proceedings.30  Here  we  present  a  summary  of  the  results  in  that  paper  and  we  attach  the 
complete  paper  in  Appendix  1 . 


Figure  2.  Each  group  of  symbols  plots  the  ROC  points  for  four  participants  in  USA-T  Study  2  and  for  one 
EEG-classifier  type.  Engagement  test  and  retest  classifiers  were  much  more  accurate  than  corresponding 
workload  classifiers.  We  simulated  the  effects  of  increasing  test  and  re-test  accuracy  of  the  workload 
classifier  by  10%  (blue  and  green  circles  vs.  purple  and  orange  triangles). 

The  paper  describes  the  use  of  multi-way  decomposition  methods  to  efficiently  summarize  EEG  data. 
A  space-frequency-time  atomic  decomposition  was  applied  to  EEG  data  recorded  while  participants 
performed  tasks  associated  with  varying  levels  of  cognitive  performance.  The  new  atomic 
decomposition  of  cognitive  workload  data  revealed  alpha  and  theta  EEG  oscillations  which  agree 
with  observations  reported  in  the  brain  research  literature.  The  temporal  signature  of  the  atoms 
discriminated  between  different  levels  of  cognitive  activity.  The  results  and  analysis  confirmed  the 
utility  of  the  multi-way  decomposition  method  to  construct  new  models  and  algorithms  for 
monitoring  cognitive  status,  which  can  supplement  or  overcome  existing  approaches  based  on 
conventional  two-dimensional  space-time  or  frequency-time  data  decomposition. 

The  results  showed  that  mental  workload  may  be  tracked  by  EEG  components  isolated  using 
PARAFAC.  Components  or  atoms  of  the  PARAFAC  decomposition  had  high  loadings  in  the 
frequency  spectrum  and  across  electrodes,  which  reflected  alpha,  theta,  and  broad-band  EEG 
processes.  Unlike  other  approaches  to  isolate  EEG  factors  related  to  mental  workload,  our  application 
of  PARAFAC  begins  with  a  truly  three-dimensional  model  of  EEG  variance  observed  during 
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performance  of  a  cognitive  task.  This  model  is  fit  to  the  EEG  simultaneously  in  frequency,  space,  and 
time.  The  time  dimension  in  the  present  study  reflects  imposed  task  demands  that  produced 
calibrated  states  of  low-or  high  mental  workload.  For  an  unsupervised  method  of  decomposition, 
these  PARAFAC  results  are  remarkable.  Using  a  small  number  of  electrodes,  the  loadings  of  several 
PARAFAC  atoms  in  time,  co- varied  with  task  demands  and  mental  workload.  Admittedly,  these 
results  are  based  on  a  small  sample  of  two  participants.  However,  the  task  performances  were  long, 
providing  about  30  minutes  of  EEG  recordings  for  analysis.  In  our  extensive  experience  with  similar 
experiments,  the  discovery  of  EEG  atoms  that  track  workload  and  have  meaningful  spectral  and 
spatial  propertied  from  recordings  of  these  durations  is  not  likely  to  arise  from  chance. 

5.2.2  USAF-C  Database  -Unpublished  Results 

5.2.2.1  Methods 

The  data  were  collected  from  six  additional  participants  in  the  USAF-C  study  (denoted  'B',  'C',  'E',  'G', 
"I',  and  'K'),  each  of  whom  completed  three  sessions  (trial  repetitions).  Participants  were  trained  to 
stable  performance  on  a  simulated  Unmanned  Air  Vehicle  (UAV)  task.  The  task  consisted  of 
monitoring  the  progress  of  four  UAVs  as  they  flew  a  preplanned  mission,  monitoring  UAV  resources, 
and  classifying  synthetic  aperture  radar  images  acquired  by  sensors  in  each  UAV.  For  additional 
details  concerning  the  experimental  design,  methods  and  procedures,  please  refer  to  Appendix  1 . 

Each  run  contained  six  different  indicators  of  mental  states  assigned  by  the  experimenters;  however, 
indicators  for  only  two  workload  levels  (low  and  high)  were  provided  to  us  due  to  a  security 
restriction.  Therefore,  we  used  only  the  data  from  the  two  periods  designated  as  high-  and  low 
workload.  Nineteen  channels  of  EEG  (placed  according  to  the  International  10-20  System31  with  a 
linked  mastoid  reference)  were  available  in  this  study.  One-channel  ECG  and  two  channels  of  bipolar 
EOG  (vertical  and  horizontal)  were  also  recorded. 

First,  the  EEG  data  were  down-sampled  to  128  Hz  sampling  rate  from  the  original  sampling  rate  of 
256  Hz.  Next,  data  were  segmented  into  non-overlapping  consecutive  windows  of  2-s  duration.  As  in 
the  initial  analyses30  (Appendix  1),  the  power  spectral  density  (PSD)  was  computed  for  each  segment 
using  the  Thomson  Multi-taper  method.32  Initial  analyses  revealed  a  high  level  of  power  at 
frequencies  below  6  Hz.  Power  at  these  frequencies  often  arose  from  motion  artifacts  and 
confounded  the  PARAFAC  analyses  of  EEG.  Therefore,  only  the  frequencies  in  the  range  of  6  to  25 
Hz  were  considered  in  this  study.  We  repeated  this  procedure  for  each  EEG  channel  separately  and 
constructed  a  three-dimensional  matrix,  A(E  x  F  x  T),  with  E  time  segments,  F  electrodes  and  PSD 
estimates  at  T  frequencies  (Eg,  1 ). 

5.2.22  Results 

5. 2. 2. 2. 1  Spectral  Representation 

The  PARAFAC  model  has  been  run  two  times.  After  the  convergence  of  the  first  run  the  points  with 
high  values  of  the  residual  variance  and  leverage,  that  is,  points  indicating  noisy  samples,  were 
inspecting.  The  points  exceeding  95  percentile  of  the  residual  variance  and  leverage  distribution  were 
removed  and  the  PARAFAC  model  was  run  again.  In  general,  this  procedure  removed  points  with 
very  high  values  of  temporal  loadings  (signatures).  The  core  consistency10  of  these  final  models  was 
in  all  cases  greater  than  85%  indicating  a  good  model  fit. 

5. 2. 2. 2. 2  Participants  B,  K 

First,  the  PARAFAC  model  was  run  using  the  full  set  of  19  EEG  electrodes.  The  results  of  the  three- 
atom  PARAFAC  model  for  Participant  B  are  depicted  in  Figure  3.  It  can  be  observed  that  the 
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temporal  signatures  of  the  second  and  third  atoms  separate  the  periods  of  the  high  and  low  workload. 
However,  high  values  of  the  loadings  vectors  at  frequencies  above  20  Hz  indicate  that  this  can  be  due 
to  the  movement  components  superimposed  to  EEG.  To  investigate  this  effect  the  spatial  loading 
vectors  for  Atom  2  and  3  are  plotted  in  Figure  4. 


1.  Atom 


Details  of  Atom2 


0  100  200  300  400  500  600  700 

Time  (s) 


Figure  3.  Loadings  of  three  PARAFAC  atoms  extracted  from  EEG  recordings  of  Participant  B  using  19 
electrodes.  Left  panel:  Temporal  signatures  of  the  EEG  atoms.  Red  marks  indicate  periods  of  high 
workload,  blue  line  marks  the  low  workload  periods.  Vertical  dotted  lines  separate  three  distinct 
experimental  sessions.  Bottom  panel:  Detailed  plot  of  temporal  loadings  of  Atom  2.  Right  panel:  Spectral 
signatures  corresponding  to  atoms  numbered  in  the  left  panel. 


Figure  4.  Participant  B.  Comparison  of  the  spatial  signatures  corresponding  to  Atom  2  and  Atom  3  plotted 
in  Figure  3. 
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High  spatial  loading  values  can  be  observed  at  T3,  T4,  F7  and  Fp2  sites  (Atom  2)  and  at  the  electrode 
sites  Fpl,  Fp2,  T5,  T6  and  F7  for  Atom  3.  These  electrodes  are  generally  known  to  be  susceptible  to 
the  movement  artifact.  Therefore  in  the  next  step  we  have  removed  these  electrodes  and  run  the 
PARAFAC  model  again.  The  results  with  the  reduced  set  of  electrodes  are  depicted  in  Figure  5. 


1 .  Atom 
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Figure  5.  Foadings  of  the  three  PARAFAC  atoms  extracted  from  EEG  recordings  of  Participant  B  using 
the  reduced  set  of  12  electrodes.  Left  panel:  Temporal  signatures  of  the  EEG  atoms.  Red  marks  indicate 
periods  of  high  workload,  blue  line  marks  the  low  workload  periods.  Vertical  dotted  lines  separate  three 
distinct  experimental  sessions.  Bottom  panel:  Detailed  plot  of  the  temporal  loadings  of  Atom  2.  Right 
panel:  Spectral  signatures  corresponding  to  the  atoms  numbered  in  the  left  panel. 


Now  three  distinct  atoms  can  be  observed.  While  the  third  atom  seems  to  represent  overall  EEG 
power  indicating  1/f  trend,  the  first  and  the  second  atom  seem  to  be  two  spectrally  complimentary 
atoms  which  when  applied  together  discriminate  periods  of  low  and  high  workload.  However, 
although  the  spectral  concentration  of  the  second  atom  around  20  Hz  and  its  decay  at  higher 
frequencies  indicate  that  this  atom  may  represent  the  beta  component  in  EEG,  the  spatial  distribution 
shows  high  influence  of  the  frontal  F3,  F4  and  F8  sites.  Therefore  the  influence  of  movement  artifact 
in  this  atom  cannot  be  ruled  out.  Note  that  using  the  reduced  set  of  electrodes  increased  the  core 
consistency  to  87%  indicating  good  fit  of  the  model  to  data.  This  is  in  contrast  when  the  full  set  of 
electrodes  was  used  and  the  core  consistency  value  below  20%  indicated  poor  fit.  Similar  three  atoms 
to  the  ones  plotted  in  Figure  5  were  observed  in  Participant  K,  however,  using  the  full  set  of 
electrodes.  While  two  atoms  resemble  Atoms  1  and  3  in  Figure  5,  the  third  extracted  atom  in  this 
participant  resemble  the  movement  atom  depicted  in  Figure  3.  This  atom  was  spatially  concentrated 
on  Fpl,  Fp2,  F8  and  T4  electrode  sites.  Removal  of  these  sites  and  also  additional  to  noise 
susceptible  sites  (F3,  F4,  F7,  F8,  T3,  T5  and  T6)  did  not  change  the  structure  of  the  extracted  atoms, 
indicating  that  the  observed  movement  related  artifact  globally  influences  EEG  recordings  at  all  sites. 

5. 2. 2. 2. 3  Participant  C 

The  three-component  PARAFAC  model  was  run  using  the  full  set  of  19  electrodes.  Two  distinct 
workload  related  atoms  (Atom  1  &  3)  can  be  observed  in  Figure  6.  While  the  spectral  signature  of  the 
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first  atom  indicates  the  importance  of  lower  (below  8Hz)  and  higher  alpha  (around  12  Hz) 
frequencies,  the  third  atom  shows  spectral  concentration  somehow  complimentary  and  peaked 
around  10  Hz.  Note  clear  separation  of  the  low  and  high  workload  levels  using  these  two  atoms. 


1 .  Atom 


3.  Atom 


Details  of  Atom  1 


Time  (s) 


Figure  6.  Foadings  of  the  three  PARAFAC  atoms  extracted  from  EEG  of  Participant  C  using  19 
electrodes.  Left  panel:  Temporal  signatures  of  the  EEG  atoms.  Red  marks  indicate  periods  of  high 
workload,  blue  line  marks  the  low  workload  periods.  Vertical  dotted  lines  separate  three  distinct 
experimental  sessions.  Bottom  panel:  The  detailed  plot  of  the  temporal  loadings  of  Atom  1 .  Right  panel: 
Spectral  signatures  corresponding  to  the  atoms  numbered  in  the  left  panel. 

Spatial  signatures  of  these  two  atoms  are  plotted  in  Figure  7.  It  can  be  observed  that  while  the  third 
atom  is  more  centro-parietal  located,  the  first  atom  indicates  higher  importance  of  the  fronto-central 
sites.  Finally,  the  second  atom  exhibits  a  typical  background  EEG  spectrum  with  exponential  decay 
of  frequencies  and  no  discrimination  of  the  high  and  low  workload  periods  can  be  observed  in  the 
corresponding  temporal  signature  plot. 
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Figure  7.  Participant  C.  Comparison  of  the  spatial  signatures  corresponding  to  Atom  1  and  Atom  3  plotted 
in  Figure  4. 


5.2.2.2A  Participants  E,  G,  I 


The  three-component  PARAFAC  model  atomic  decomposition  has  revealed  almost  identical  spatial, 
spectral  and  temporal  structure  of  the  loading  vectors  in  these  three  participants.  The  loading  vectors 
are  very  similar  the  ones  plotted  in  Figure  5,  but  in  this  case  the  full  set  of  19  electrodes  was  used. 
Three  distinct  atoms  observed  in  these  participants  are  plotted  in  Figure  8  (Participant  E). 


1.  Atom 
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Figure  8.  Loadings  of  the  three  PARAFAC  atoms  extracted  from  EEG  recordings  of  Participant  E  using 
the  full  set  of  19  electrodes.  Left  panel:  Temporal  signatures  of  the  EEG  atoms.  Red  marks  indicate 
periods  of  high  workload,  blue  line  marks  the  low  workload  periods.  Vertical  dotted  lines  separate  three 
distinct  experimental  sessions.  Bottom  panel:  The  detailed  plot  of  the  temporal  loadings  of  Atom  2.  Right 
panel:  Spectral  signatures  corresponding  to  the  atoms  numbered  in  the  left  panel. 

While  the  first  atom  resembles  the  movement  related  atom  (Atom  2)  already  observed  in  Participant 
B  (spatially  concentrated  around  T3  and  T4  sites),  the  spectral  distribution  of  the  second  atom 
resembles  Atom  3  observed  in  Participant  C.  However,  the  spatial  distribution  of  this  atom  is 
different  and  high  loading  values  are  concentrated  in  occipital  sites  Ol  and  02  (clearly  visible  in 
Figure  9  for  Participants  E  and  I).  This  spatial  distribution  was  similar  for  all  three  Participants  E,  G 
and  I.  However,  the  spectral  peak  in  Participants  G  and  I  was  concentrated  a  bit  higher  at  1 1  Hz. 

Note  the  opposite  pattern  in  the  temporal  signature  of  Atom  2  in  comparison  to  Atom  3  in  Participant 
C.  Now,  the  values  of  high  workload  periods  are  above  the  values  indicating  the  low  workload 
periods.  The  third  atom  seems  to  represents  the  mixture  of  a  typical  background  EEG  spectrum  with 
increased  concentration  in  lower  than  8  Hz  frequencies.  In  Participants  G  and  I  this  concentration  of 
higher  loadings  values  on  lower  frequencies  was  not  observed  and  a  clear  1/f  spectral  pattern  could 
be  observed.  Partial  separation  of  the  high  and  low  workload  periods  can  be  also  observed  in  the 
temporal  signature  of  this  atom  for  Participant  E.  This  was  not  rue  for  Participants  G  and  I  where  this 
background  EEG  atom  has  shown  no  discrimination  abilities. 
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Figure  9.  Comparison  of  the  spatial  signatures  corresponding  to  the  workload  atom  (Atom  2  in  Figure  6) 
for  participant  E,  G  and  I. 


5. 2. 2. 2. 5  Coherence  Representation 


From  the  PSDs  we  also  computed  three  measures  of  coherence  the  magnitude  squared  coherence,  or 
CV1.  the  cross  -spectral  power  density  via  Welch's  method,  or  Pxv,  and  the  transfer  function  estimate 
using  Welch's  averaged  periodogram  method,  or  Txy: 


Cjf)= 


\pfft 


(Eq.  1) 


00 

m — — go 


(Eq.  2) 


(Eq.  3) 


Where  Rxy  is  the  cross-correlation  sequence  of  the  EEG  segments  from  channels  x  and  y,  or 

RJm)=E\  Xn+my*n  } 1 =  E{  Xnin-m  }  (Efi-  4) 


A  three-component  PARAFAC  model  has  shown  a  good  fit  to  data.  For  individual  participants  the 
same  EEG  electrodes  were  used  as  in  the  case  of  spectral  representation.  Coherence  among  all 
possible  electrode  pairs  was  computed  and  used  in  the  model. 

5. 2. 2. 2. 6  Participants  B,  E,  G,  I,  K 

Interestingly,  the  PARAFAC  model  trained  on  data  using  the  fullest  of  19  electrodes  provided  the 
results  without  two  movement  related  atoms  in  contrast  to  the  case  when  the  spectral  representation 
was  used  (Figure  3).  The  extracted  atoms  using  the  coherence  representation  are  plotted  in  Figure 
10.  Pxy  and  Txy  representation  resulted  in  atoms  with  similar  spectral  signatures  but  temporal 
signatures  were  less  discriminative  regarding  the  two  different  periods  of  low  and  high  workload. 
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Frequency(Hz)  Time(s) 


Figure  10.  Loadings  of  the  three  PARAFAC  atoms  extracted  from  EEG  recordings  of  Participant  B  using 
the  full  set  of  19  electrodes.  Top-Left  panel:  Spectral  signatures.  Top-right  panel:  Z-scores  of  the 
temporal  signatures  of  the  EEG  atoms  numbered  in  the  right  panel.  Shaded  areas  indicate  periods  of  low 
workload.  For  the  Z-scores  computation  (mean  and  standard  deviation)  data  of  low  workload  periods 
were  used.  This  was  done  for  each  trial  separately.  Vertical  dotted  lines  separate  two  distinct  experimental 
sessions.  Moving  average  smoothing  with  the  length  of  60  points  was  used.  Bottom  panel:  The  coherence 
plot  (connecting  lines)  of  the  electrode  pairs  exceeding  the  90  percentile  of  the  all  coherence  values 
distribution  (Note,  the  label  Factor  in  the  plots  means  the  same  as  Atom). 


In  Figure  10,  Atom  3  with  a  peaked  spectral  concentration  around  10  Hz  and  concentration  at  lower 
than  7  Hz  can  be  observed.  This  evident  peaked  spectral  concentration  is  in  contrast  to  the  case  of 
using  the  spectral  representation  (Figure  5,  Atom  1).  The  temporal  signature  of  this  atom  nicely 
separates  the  periods  of  low  and  high  workload.  The  value  of  90  percentile  was  computed  from  the 
distribution  of  the  spatial  loading  vector  of  electrode  pairs  and  all  pairs  for  which  the  loading  values 
exceeded  this  percentile  (value  equal  to  0.1032)  are  plotted  in  the  bottom  part  of  Figure  10.  The  plot 
suggests  the  importance  of  the  parieto-ocipital  and  ocipito-ocipital  electrode  pairs  for  this  workload 
discriminative  atom  (Atom  3).  The  spatial  distribution  of  Atom  1  in  the  frontal  and  fronto-temporal 
sites  suggests  a  movement  related  component  in  the  EEG  and  somehow  ruling  out  the  fact  that  this 
atom  represents  an  EEG  beta  band  related  component.  Finally,  Atom  2  represents  the  general  EEG 
component  with  the  decaying  spectral  characteristic  and  regular  spatial  distribution  over  the  whole 
brain.  Very  similar  spatial,  temporal  and  spectral  signatures  to  Participant  B  where  observed  in 
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Participants  E,  G,  K,  and  I.  For  example,  in  Figure  1 1  the  results  for  Participant  I  are  plotted.  In 
Figure  12  spatial  and  spectral  signatures  for  the  workload  related  atom  (Atom  3  in  Figure  10  or  in 
Figure  1 1)  are  compared.  Close  match  can  be  observed  in  the  spatial  distribution  of  the  electrode 
pairs  among  all  five  participants  (left  plot).  Although,  some  variation  in  the  location  of  the  spectral 
peak  can  be  observed  in  the  right  plot  of  Figure  12,  the  overall  pattern  of  this  spectral  signature 
seems  to  be  consistent  over  all  five  participants. 


Frequency(Hz)  Time(s) 


Figure  1 1 .  Eoadings  of  the  three  PARAFAC  atoms  extracted  from  EEG  recordings  of  Participant  I  using 
the  full  set  of  19  electrodes.  Top-Left  panel:  Spectral  signatures.  Plotting  conventions  are  the  same  as  in 
Figure  10. 

5. 2. 2. 2. 7  Participant  C 

The  atomic  decomposition  using  the  Cxy  coherence  data  representation  is  depicted  in  Figure  13.  We 
can  observe  three  very  distinct  atoms  which  resemble  the  atoms  in  Figure  7  when  spectral 
representation  was  used.  However,  more  compact  (peaked)  spectral  representation  for  Atom  1  and  3 
can  be  observed  now.  We  can  clearly  see  that  Atom  1  has  the  spectral  peak  at  13  Hz  and  almost  zero 
loadings  values  at  8  Hz.  Atom  3  is  peaked  at  10  Hz  and  a  small  spectral  bump  can  be  observed  over 
the  beta  band  (16  to  22  Hz).  Comparing  the  temporal  plots  (not  plotted  here),  the  coherence 
representation  also  provides  visually  clearer  separation  of  the  low  and  high  workload  periods  than 
can  be  observed  in  Figure  7  (this  is  specifically  true  for  Atom  1).  Coherence  plots  in  Figure  13 
indicates  the  importance  the  fronto-central  and  centro-parietal  pairs  with  some  importance  of  the 
local  coherences  around  midline  of  all  frontal,  central,  parietal,  and  also  occipital  sites.  For  Atom  3 
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the  high  coherence  values  were  observed  for  the  parieto-occiptal  pairs.  Fower  loadings  were 
observed  between  frontal  channels  and  for  the  electrode  pairs  connecting  to  the  T6  electrode  site.  We 
do  not  show  results  for  the  Pxy  and  Txy  measures,  which  appeared  physiologically  implausible. 


Figure  12.  Participants  B,  E,  G,  I,  and  K.  Left:  Comparison  of  the  spatial  (electrode  pairs)  signatures 
(loading  vectors)  corresponding  to  the  workload  atom.  Right:  Comparison  of  the  spectral  signatures 
(loading  vectors)  corresponding  to  the  workload  atom.  Examples  of  this  workload  atom  are  plotted  in 
Figure  10,  Atom  3,  for  Participant  B  and  in  Figure  11,  Atom  3,  for  Participant  I. 


Frequency(Hz)  Time(s) 


Figure  13.  Loadings  of  the  three  PARAFAC  atoms  extracted  from  EEG  recordings  of  Participant  C  using 
the  full  set  of  19  electrodes.  Plotting  conventions  are  the  same  as  in  Figure  10. 
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5.2.3  USA-T  Database 
5.2.3. 1  Methods 

The  USA-T  database  was  derived  from  a  prior  ARO-funded  SBIR  Phase  II  contract  awarded  to 
Quantum  Applied  Science  and  Research.2  Under  a  subcontract,  PDT  supported  the  studies 
performed  under  this  contract,  including  experimental  design,  pre-processing  data  and  developing 
algorithms  for  estimation  and  monitoring  mental  workload  using  PLS  methods,  reporting,  and 
publications.  The  prime  contractor  has  licensed  our  use  of  the  data  from  the  original  ARO  SBIR 
contract  for  further  research.33  The  studies  consisted  of  a  series  of  experiments  in  which  we 
established  increasing  control  over  task,  performance,  and  physiological  factors.  The  first  two 
experiments  primarily  served  to  evaluate  the  major  effects  of  workload  on  performance  and 
physiological  variables  and  to  compare  two  candidate  tasks  for  the  third  experiment. 

A  detailed  description  of  experimental  and  analytical  methods  for  the  prior  study,  from  which  the 
current  study  data  were  obtained,  has  been  submitted  for  publication  and  is  available  on  line6. 

Briefly,  data  were  collected  from  16  participants  recruited  from  the  San  Francisco  Bay  Area 
community.  The  participants  included  12  males  and  4  females  with  a  mean  age  of  26.9  (SD  =  7.4) 
years.  Subjective  moods  were  indexed  by  the  Activation  Deactivation  Adjective  Checklist 34  (AD- 
ACL)  and  the  Visual  Analogue  Mood  Scales35  (VAMS)  questionnaires.  Observed  behavior  included 
ratings  of  activity  and  alertness  from  videotaped  recordings  of  each  participant’s  performance.  The 
performance  measures  were  response  time  and  response  accuracy.  The  physiological  measures  were 
derived  from  spontaneous  EEGs  and  EOGs. 

For  the  present  analyses,  we  are  concerned  only  with  data  from  Experiment  2,  in  which  participants 
performed  a  first  person  shooter  simulation  (Battlefield  II)  at  three  difficulty  levels:  one  passive  level 
(observe  only)  and  two  active  levels,  easy  (1  enemy)  and  difficult  (5  enemies).  A  secondary  auditory 
tone-report  task  was  used  to  obtain  independent  evidence  of  high  workload  states.  The  experimental 
design  crossed  the  two  auditory  task  levels  (attend,  ignore)  with  the  two  active  difficulty  levels.  In 
the  attend  tones  condition,  participants  were  instructed  to  silently  tally  the  numbers  of  kills,  deaths, 
and  tones,  press  a  foot  pedal  to  acknowledge  the  tones  as  quickly  as  possible,  and  report  the  tallies 
after  each  pedal  press.  The  modified  tone-report  task  served  to  provide  a  time-indexed  measure  of  the 
workload  imposed  by  the  game  difficulty  manipulation.  Because  the  tone-report  task  was  not  as  time 
critical  as  simulated  combat,  we  hypothesized  that  reaction  times  to  the  tones  would  be  greater 
during  high  workload  periods  than  during  low  workload  periods.  A  spectral  analysis  of  EEG  activity 
was  performed  for  each  condition.  Six  participants  performed  the  tasks.  Of  these  the  first  two  were 
for  determination  of  measurements  and  procedures.  The  remaining  four  participants  completed  all 
conditions.  We  also  asked  each  participant  to  complete  the  NASA  TLX  workload  rating  scale  after 
he  or  she  completed  each  game  condition. 

Here  we  have  analyzed  data  from  the  four  participants  (A,  B,  C,  D)  who  completed  all  conditions. 
Each  participant  performed  two  sessions  (or  repetitions  of  all  conditions)  on  two  separate  days.  For 
an  outline  of  the  protocol  for  sessions  and  days  see  Appendix  2 

We  recorded  EEG  from  a  standard  19-channel  referential  montage  placed  on  the  scalp  according  to 
the  International  10-20  system.  EEGs  were  sampled  at  128  Hz  with  an  analog  low  pass  filter  setting 
of  64  Hz.  No  additional  filtering  was  applied  to  data.  EEGs  were  then  segmented  into  consecutive 
non-overlapping  2-s  windows.  As  in  the  USAF-C  analyses,  we  used  the  Thompson  Multi-taper 
method  to  compute  power  spectral  densities  (PSD).  From  the  PSDs  we  also  computed  three  measures 
of  coherence,  the  magnitude  squared  coherence,  or  Cxy_  the  cross  -spectral  power  density,  or  Pxv,  and 
the  transfer  function  estimate,  or  TnK  as  defined  in  Equations  2-5  above. 
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5.2.3.2  Results 

5. 2. 3. 2.1  Spectral  representation 

In  the  first  step,  a  spectral  representation  of  the  EEG  was  used  in  the  PARAFAC  model.  First,  the 
PARAFAC  model  was  run  using  the  full  set  of  19  EEG  electrodes.  This  was  done  by  using  the  data 
from  each  day  and  each  participant  separately.  However,  the  vector  of  electrode  loadings  with  very 
high  levels  indicated  a  noise  component  on  some  electrode  sites.  Very  high  noise  levels  were  in 
general  observed  at  the  temporal  sites  T7  and  T8  and  the  frontal  sites  Fpl,  Fp2,  Fz,  F4  or  F5.  This 
was  already  observed  in  the  Quasar  study.  Therefore,  the  electrodes  indicating  high  level  of  noise 
were  selectively  removed  and  the  PARAFAC  model  was  run  again.  This  was  done  for  each 
participant  separately.  In  all  the  case  this  resulted  into  the  extraction  of  the  physiologically  more 
plausible  results  in  comparison  to  the  models  with  the  full  EEG  electrodes  set.  In  addition  the 
PARAFAC  model  has  been  run  two  times.  After  the  convergence  of  the  first  run  the  points  with  high 
values  of  the  residual  variance  and  leverage,  that  is,  points  indicating  noisy  samples,  were  inspecting. 
The  points  exceeding  95  or  99  percentile  of  the  residual  variance  and  leverage  distribution  were 
removed  and  the  PARAFAC  model  was  run  again.  In  general,  this  procedure  removed  points  with 
very  high  values  of  temporal  signatures.  The  core  consistency10  of  these  final  models  was  in  all  cases 
greater  than  85%  indicating  a  good  model  fit. 

5. 2. 3. 2. 1.1  Participant  A 

After  running  the  PARAFAC  model  with  the  full  set  of  electrodes,  the  electrodes  located  at  sites  Fz, 
F4,  T7,  T8,  and  Fp2  were  removed  from  the  further  analysis.  Figure  14  summarizes  the  results 
obtained  by  running  the  PARAFAC  model  on  the  first  day  data.  Three  distinct  atoms  can  be  observed. 
While  the  first  atom  seems  to  represent  overall  EEG  power,  the  second  and  the  third  atoms 
discriminate  workload  increase  and  engagement/disengagement  in  the  task.  While  the  spectral 
signature  vector  of  the  second  atom  shows  two  peaks  in  the  theta  and  alpha  band,  the  spectral 
signature  of  the  third  atom  has  a  single  peak  in  lower  alpha  band.  Note  that  these  two  spectral  loading 
vectors  have  complimentary  character.  The  bottom  plot  shows  in  details  the  temporal  signature  of  the 
second  atom  where  a  nice  separation  of  the  individual  workload  levels  can  be  observed.  Note  that  for 
better  visualization  the  values  of  the  temporal  loading  vectors  were  smoothed  with  a  robust  quadratic 
fit  smoothing  method  using  the  smoothing  window  of  10  points. 
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Figure  14.  Loadings  of  the  three  PARAFAC  atoms  extracted  from  EEG  recordings  of  Participant  A,  Day 
1.  Left  panel:  Temporal  signatures  of  the  EEG  atoms.  Colors  indicate  periods  of  different  workload  tasks 
(red  -  resting/relaxing;  black  -  1  enemy;  green  -  1  enemy  +  counting;  purple  -  5  enemies;  cyan  -  5  enemies 
+  counting).  Vertical  dotted  line  separates  two  distinct  experimental  sessions.  Bottom  panel:  The  detailed 
plot  of  the  temporal  loadings  of  Atoms  2.  Right  panel:  Spectral  signatures  corresponding  to  the  atoms 
numbered  in  the  left  panel. 

However,  it  is  interesting  to  see  temporal  signatures  without  smoothing,  especially  during  the 
transition  periods  between  different  workload  tasks.  This  is  depicted  in  Figure  15  for  three  different 
workload  tasks.  We  can  see  very  nice  decrease  during  these  transient  periods  when  the  protocol  is  re¬ 
set  and  participant  is  relaxed.  The  temporal  signature  values  of  these  transition  periods  are  similar  to 
task  1  when  the  participants  were  instructed  to  be  relaxed. 
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Figure  15.  Transient  periods  between  three  different  workload  task  (1  enemy  +  counting;  5  enemies;  5 
enemies  +  counting).  No  smoothing  of  the  loadings  values  was  used. 


Next,  the  same  PARAFAC  model  setting  was  used  for  the  second  day  data  of  the  same  participant. 
The  results  are  depicted  in  Figure  16.  Again  we  can  see  two  distinct  atoms  reflecting  workload  levels 
(Atom  1)  and  engagement/disengagement  (Atom  3).  Although  the  Atom  1  does  not  show  as  clear 
peaks  as  in  the  case  of  Day  1  data,  high  levels  of  low  frequency  (theta)  and  an  increase  in  the  higher 
alpha  is  visible.  Atom  3  reflecting  engagement/disengagement  seems  to  be  stable  over  the  both  days. 
This  stability  can  be  also  observed  in  the  spatial  signatures  plotted  in  Figure  17.  We  can  see  that  this 
‘engagement/disengagement  atom’  is  associated  with  high  loadings  in  midline  parieto-central  and 
occipital  sites.  The  spatial  loading  vectors  for  the  ‘workload  atom’  are  depicted  in  Figure  18.  In  this 
case  the  high  spatial  loading  levels  can  be  observed  in  the  occipital  region  and  over  the  P7  and  P8 
regions.  The  high  values  over  the  F7  and  Fpl  observed  during  the  first  day  were  not  observed  in  the 
second  day  results. 

Next,  the  stability  and  predictability  of  these  models  were  investigated  by  varying  training  data  sets. 
First,  the  model  was  trained  using  the  Day  1  data  and  the  temporal  signature  corresponding  to  the 
workload  atom  was  extracted  using  the  second  day  data.  This  signature  was  then  compared  with  the 
temporal  signature  extracted  using  the  second  day  data  only.  Then,  the  procedure  was  repeated  by 
switching  the  days;  that  is,  the  model  was  constructed  using  the  second  day  data  and  the  workload 
atom  related  temporal  signature  from  the  first  day  data  was  extracted.  The  results  of  this  experiment 
are  depicted  in  Figure  19  and  Figure  20.  We  can  see  very  close  match  between  the  predicted 
signatures  atoms  and  the  signatures  extracted  directly  from  the  data.  This  is  very  promising  result 
indicating  ability  of  the  method  to  extract  stable  data  components  which  can  be  used  for  predicting 
workload  levels  from  the  data  recorded  at  in  time  separated  runs  (days).  The  same  stable  temporal 
signatures  reflecting  workload  levels  were  also  observed  in  participant  C. 
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Figure  16.  Loadings  of  the  three  PARAFAC  atoms  extracted  from  EEG  recordings  of  Participant  A,  Day 
2.  Left  panel:  Temporal  signatures  of  the  EEG  atoms.  Plotting  conventions  are  the  same  as  in  Figure  14. 
Bottom  panel:  The  detailed  plot  of  the  temporal  loadings  of  Atom  1.  Right  panel:  Spectral  signatures 
corresponding  to  the  atoms  numbered  in  the  left  panel. 


Figure  17.  Participant  A.  Comparison  of  the  Day  1  and  Day  2  spatial  signatures  (loading  vectors) 
corresponding  to  the  engagement/disengagements  atom. 
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Figure  18.  Participant  A.  Comparison  of  the  Day  1  and  Day  2  spatial  signatures  (loading  vectors) 
corresponding  to  the  workload  atom. 


Time  (s) 

Figure  19.  Temporal  signatures  for  Day  2.  Comparison  of  the  predicted  temporal  signature  of  the 
workload  atom  using  the  PARAFAC  model  trained  on  Day  1  data  and  the  temporal  signature  (orange) 
extracted  from  the  Day  2  PARAFAC  model.  Colors  indicate  periods  of  different  workload  tasks  (red  - 
resting/relaxing;  black  -  1  enemy;  green  -  1  enemy  +  counting;  purple  -  5  enemies;  cyan  -  5  enemies  + 
counting). 
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Figure  20.  Temporal  signatures  for  Day  1.  Comparison  of  the  predicted  temporal  signatue  of  the  workload 
atom  using  the  PARAFAC  model  trained  on  Day  2  data  and  the  temporal  signature  (orange)  extracted 
from  the  Day  1  PARAFAC  model.  Colors  indicate  periods  of  different  workload  tasks  (red  - 
resting/relaxing;  black  -  1  enemy;  green  -  1  enemy  +  counting;  purple  -  5  enemies;  cyan  -  5  enemies  + 
counting). 

5. 2. 3. 2. 1.2  Participant  B 

Running  the  three  components  PARAFAC  model  in  this  participant  resulted  in  a  low  (<  40%)  core 
consistency  value  indicating  a  poor  fit  of  the  model.  This  was  in  spite  of  the  fact  that  the  EEG 
channels  (Fz,  F7,  F8,  T7,  T8,  and  Fp2)  showing  a  high  level  of  noise  was  removed  after  running  the 
PARAFAC  model  using  the  full  set  of  19  electrodes.  Therefore,  two  components  model  was  used. 

The  results  of  this  model  for  the  first  day  are  depicted  in  Figure  21.  Similar  results  were  obtained  for 
Day  2  and  are  not  plotted  here.  Again,  we  can  observed  the  temporal  signatures  reflecting  the 
participant’s  engagement/disengagement  in  the  task  (Atom  1),  but  no  clear  atom  reflecting  the 
workload  levels  can  be  observed.  The  spatial  signatures  of  Atom  1  were  very  similar  to  ones 
observed  in  participant  A;  that  is,  signatures  with  high  values  over  midline  parietal  and  occipital  sites 
were  observed. 

5. 2. 3. 2. 1.3  Participant  C 

Due  to  high  level  of  noise  channels  Fz,  F3,  F4,  Fpl,  Fp2,  T7  and  T8  were  removed  from  the  analysis. 
Three  components  PARAFAC  model  has  shown  three  distinct  atoms  and  the  results  also  indicated 
good  fit  of  the  model  to  the  data  (core  consistency  >  85%).  Results  using  the  Day  1  and  Day  2  data 
are  depicted  in  Figure  22  and  Figure  23,  respectively.  Again  we  can  clearly  see  the  temporal 
signatures  discriminating  participant’s  engagement  and  disengagement  in  the  task  (Atom  3,  Day  1 
and  Atom  3,  Day  2).  This  atom  has  spectral  peak  at  about  8  Hz  and  its  spatial  distribution  resemble 
the  one  already  observed  in  Participants  A  and  B;  that  is,  a  higher  loadings  can  be  observed  over  the 
mid-line  centro-parietal  and  occipital  regions.  Again,  similar  to  participant  A,  the  workload  related 
temporal  signatures  can  be  observed  (Atom  2,  Day  1,  and  Atom  3,  Day  2). 
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Figure  21.  Loadings  of  the  two  PARAFAC  atoms  extracted  from  EEG  recordings  of  Participant  B,  Day  1. 
Left  panel:  Temporal  signatures  of  the  EEG  atoms.  Plotting  conventions  are  the  same  as  in  Figure  14. 
Bottom  panel:  The  detailed  plot  of  the  temporal  loadings  of  Atom  1.  Right  panel:  Spectral  signatures 
corresponding  to  the  atoms  numbered  in  the  left  panel. 

The  spatial  distribution  of  this  workload  related  atom  reveled  higher  loading  values  over  the  occipital 
region,  but  in  addition,  and  in  contrast  to  participant  A,  higher  values  of  the  loading  vectors  were  also 
observed  over  the  midline  central  and  parietal  sites  (Cz,  Pz).  Similar  to  participant  A,  testing  for  stability 
and  predictability  revealed  highly  consistent  extraction  of  the  temporal  signatures  when  training  and 
testing  sets  were  varied;  that  is,  building  the  model  on  one  day  and  extracting  the  temporal  signatures 
using  the  other  day  data.  Finally,  in  contrast  to  participant  A  and  B,  the  third  temporal  signature  indicates 
possibility  to  discriminate  periods  when  participant  were  instructed  to  count  events  versus  the  periods 
without  counting.  This  atom  (Atom  1  in  Figure  22  and  Figure  23)  shows  clear  peak  centered  around  10 
Hz  and  its  spatial  distribution  shows  concentration  over  the  parietal  sites  (Pz,  P3,  P4).  Note  that  the 
counting  periods  are  somehow  mixed  with  the  periods  of  disengagement  (red).  This  can  be  due  to  the  fact 
that  the  engagement/disengagement  atom  is  also  characterized  by  high  values  of  spatial  loadings  over  the 
parietal  sites  (see  Figure  17).  However,  in  contrast  to  the  ‘counting/non-counting’  atom,  the 
engagement/disengagement  atom  has  the  spectral  peak  shifted  to  lower  frequencies  and  centered  at 
around  8  Hz.  Interestingly,  the  combination  of  these  two  atoms  suggests  the  ability  of  a  hierarchical 
discrimination  between  engagement  and  disengagement  in  the  first  step,  and  in  the  second  step  the 
discrimination  between  counting  and  no  counting  periods  when  the  participant  in  engaged  in  the  task. 
Note  that  this  is  very  interesting  result  not  observed  in  the  previous  studies.  However,  until  now  this 
atomic  decomposition  was  observed  only  in  participant  C. 

5. 2. 3. 2. 1.4  Participant  D 

Channels  F7,  F8,  T7,  T8,  Fpl  and  Fp2  were  removed  after  running  the  PARAFAC  model  with  the 
full  set  of  19  EEG  electrodes.  Results  for  the  three  components  PARAFAC  model  computed  over  the 
first  day  data  are  depicted  in  Figure  24.  Again  the  first  and  the  second  atoms  represent  the 
engagement/disengagement  and  workload  atoms  already  observed  in  the  previous  participants. 
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Figure  22.  Loadings  of  the  three  PARAFAC  atoms  extracted  from  EEG  recordings  of  Participant  C,  Day 
1.  Left  panel:  Temporal  signatures  of  the  EEG  atoms.  Plotting  conventions  are  the  same  as  in  Figure  14. 
Bottom  panel:  The  detailed  plot  of  the  temporal  loadings  of  Atom  2.  Right  panel:  Spectral  signatures 
corresponding  to  the  atoms  numbered  in  the  left  panel. 
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Figure  23.  Loadings  of  the  three  PARAFAC  atoms  extracted  from  EEG  recordings  of  Participant  C,  Day 
2.  Left  panel:  Temporal  signatures  of  the  EEG  atoms.  Plotting  conventions  are  the  same  as  in  Figure  14. 
The  detailed  plot  of  the  temporal  loadings  of  Atom  3.  Right  panel:  Spectral  signatures  corresponding  to 
the  atoms  numbered  in  the  left  panel. 
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Figure  24.  Foadings  of  the  three  PARAFAC  atoms  extracted  from  EEG  recordings  of  Participant  D,  Day 
1 .  I.el't  panel:  Temporal  signatures  of  the  EEG  atoms.  Colors  indicate  periods  of  different  workload  tasks 
(red  -  resting/relaxing;  black-  1  enemy;  green-  1  enemy  +  counting;  purple  -  5  enemies;  cyan  -  5  enemies 
+  counting).  Vertical  dotted  line  separates  two  distinct  experimental  sessions.  Bottom  panel:  The  detailed 
plot  of  the  temporal  loadings  of  Atom  2.  Right  panel:  Spectral  signatures  corresponding  to  the  atoms 
numbered  in  the  left  panel. 


Also  the  spatial  distribution  resembles  the  previously  observed  patterns  of  these  two  distinct  atoms 
and  is  not  plotted  here.  However,  unexpectedly  low  values  (drops)  of  the  temporal  signatures  can  be 
observed  during  several  time  periods.  Detailed  investigation  of  the  PSD  has  revealed  4  to  5  times 
higher  power  over  all  EEG  electrodes,  predominately  in  the  lower  frequency  bands,  indicating 
severe  noise  component  possibly  due  to  the  measuring  device  failure.  Thus,  the  proposed  atomic 
decomposition  method  also  demonstrates  ability  to  automatically  detect  noisy  periods,  and  avoid  the 
tedious  task  of  the  visual  inspection  of  individual  2-sec  periods  one  by  one.  Finally,  results  using  the 
second  day  data  are  plotted  in  Figure  25.  Again  spectral  and  spatial  (not  plotted  here)  signatures 
indicate  the  presence  of  the  engagement/disengagement  (Atom  1)  and  the  workload  atom  (Atom  3). 
However,  these  atoms  seem  to  be  somehow  mixed.  The  spectral  peak  of  the 
engagement/disengagement  atom  is  more  spread  to  higher  frequencies  and  not  compactly  peaked  as 
we  could  observe  in  the  participants  A,  B  and  C.  Also  the  temporal  signatures  does  not  indicate  as 
high  discrimination  abilities  between  the  engagement  and  disengagement  periods  and  workload 
periods  as  we  could  observe  especially  in  the  participants  A  and  C. 

5.23.2.2  Coherence  representation 

Again,  except  the  participant  B  a  three-  component  PARAFAC  model  has  shown  a  good  fit  to  data. 
In  the  case  of  participant  B  two  components  model  was  preferred.  For  individual  participants  the 
same  EEG  electrodes  were  used  as  in  the  case  of  spectral  representation.  Coherence  among  all 
possible  electrode  pairs  was  computed  and  used  in  the  model. 
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Figure  25.  Loadings  of  the  three  PARAFAC  atoms  extracted  from  EEG  recordings  of  Participant  D,  Day 
2.  Left  panel:  Temporal  signatures  of  the  EEG  atoms.  Colors  indicate  periods  of  different  workload  tasks 
(red  -  resting/relaxing;  black-  1  enemy;  green-  1  enemy  +  counting;  purple  -  5  enemies;  cyan  -  5  enemies 
+  counting).  Vertical  dotted  line  separates  two  distinct  experimental  sessions.  Bottom  panel:  The 
detailed  plot  of  the  temporal  loadings  of  Atom  3.  Right  panel:  Spectral  signatures  corresponding  to  the 
atoms  numbered  in  the  left  panel. 

5.23.2.2.1  Participant  A 

The  PARAFAC  model  with  the  coherence  representation  resulted  in  the  extraction  of  three 
distinct  atoms  (for  Day  1,  Figure  26).  These  atoms  resemble  the  atoms  observed  when  spectral 
representation  was  used  (Figure  14).  However,  the  coherence  representation  indicates  a  compactly 
peaked  coherence  at  around  8  Hz  (more  narrow)  for  the  engagement/disengagement  atom  (Atom  3) 
and  a  smooth  frequency  loadings  representation  for  the  workload  atom  (Atom  2).  Similar  results 
were  observed  for  the  second  day  data  and  are  not  plotted  here.  The  value  of  90  percentile  was 
computed  from  the  distribution  of  the  spatial  loading  vector  of  electrode  pairs  and  all  pairs  for  which 
the  loading  values  exceeded  this  percentile  (value  equal  to  0.1307)  were  connected  in  the  bottom  plot 
of  Figure  26.  The  spatial  distribution  of  the  workload  atom  (Atom  2)  indicates  importance  of  the 
parieto-ocipital,  occipito-occipital,  but  also  some  influence  of  the  parieto-central  and  front-frontal 
pairs.  The  same  procedure  was  repeated  for  Atom  3  representing  the  engagement/disengagement. 
Ninety  percentile  was  equal  to  0.161 1  and  extracted  coherence  pairs  indicating  mainly  the 
importance  of  the  centro-frontal,  centro-central,  and  centro-parietal  pairs.  The  influence  of  the  fronto- 
frontal  and  fronto-central  pairs  can  be  also  observed  but  coherence  values  for  these  electrode  pairs 
were  smaller.  Next,  in  Figure  27  the  distribution  of  the  spatial  signatures  representing  importance  of 
the  individual  coherence  electrode  pairs  is  compared  for  Day  1  and  Day  2.  Very  good  match  between 
these  two  days  can  be  observed  for  both  the  workload  and  engagement/disengagement  atoms 
indicating  stability  of  these  components  over  two  distinct  days. 
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Figure  26.  Foadings  of  the  three  PARAFAC  atoms  extracted  from  EEG  recordings  of  Participant  A,  Day 
1.  Top-Left  panel:  Spectral  signatures.  Top-right  panel:  Z-scores  of  the  temporal  signatures  of  the  EEG 
atoms  numbered  in  the  right  panel.  Shaded  areas  indicate  periods  of  different  workload  (from  left  to  right: 
(resting/relaxing;  1  enemy;  1  enemy  +  counting;  5  enemies;  5  enemies  +  counting).  For  the  Z-scores 
computation  (mean  and  standard  deviation)  data  of  low  workload  periods  were  used.  This  was  done  for 
each  trial  separately.  Vertical  dotted  line  separates  two  distinct  experimental  sessions.  Moving  average 
smoothing  with  the  length  of  60  points  was  used.  Bottom  panel:  The  coherence  plot  (connecting  lines)  of 
the  electrode  pairs  exceeding  the  90  percentile  of  the  all  coherence  values  distribution. 


In  contrast  to  Cxy,  the  Pxy  and  Txy  coherence  measures  did  not  reveal  clear  discriminative  power  of  the 
temporal  signatures  corresponding  to  the  engagement/disengagement  and  workload  atoms.  However, 
a  distinct  atom  separating  the  periods  of  engagement  and  disengagement  was  observed  in  the  case  of 
Txy  during  the  both  days.  The  vectors  of  temporal,  spatial  and  frequency  signatures  of  this  atom  are 
plotted  in  Figure  28.  High  values  of  loadings  concentrated  at  frequencies  above  19  Hz  can  be 
observed.  This  suggest  the  importance  of  the  beta  frequencies  but  also  the  possibly  of  a  noise 
component  due  to  the  movement.  Note  that  this  atom  nicely  separates  engagement  periods  versus 
disengagements  periods  also  when  participants  are  shortly  relaxed  during  the  periods  when  the 
workload  task  are  reset.  To  better  understand  this  atom,  the  spatial  distribution  of  the  electrode  pairs 
was  investigated.  It  can  be  observed  that  the  dominate  coherence  pairs  are  almost  all  connected  to  P8, 
therefore  indicating  possible  movement  related  component  in  EEG  which  helps  to  separate  the 
periods  on  task  engagement  versus  disengagement.  This  is  important  finding  because  one  can 
possibly  filter  out  this  component  if  this  movement  related  atom  is  not  of  interest. 
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Figure  27.  Participant  A.  comparison  of  the  Day  1  and  Day  2  spatial  (electrode  pairs)  signatures  (loading 
vectors)  corresponding  to  the  workload  (left)  and  engagement/disengagements  atoms  (right). 


Figure  28.  Loadings  of  the  ‘movement  related’  PARAFAC  atom  (Atom  2)  extracted  from  EEG 
recordings  of  Participant  A,  Day  2.  Top-Left  panel:  Spectral  signatures.  Top-right  panel:  Z-scores  of  the 
temporal  signatures  of  the  EEG  atoms  numbered  in  the  right  panel.  Plotting  conventions  are  the  same  as 
in  Figure  26.  Bottom  panel:  The  coherence  plot  (connecting  lines)  of  the  electrode  pairs  exceeding  the  90 
percentile  of  the  all  coherence  values  distribution. 
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5. 2. 3. 2. 2. 2  Participant  B 

Similar  to  the  spectral  representation,  the  coherence  representation  did  not  reveal  the  workload 
related  atom  in  this  participant.  The  engagement/disengagement  atom  has  shown  more  compact 
spectral  profile  (narrow)  peak  around  1 1  Hz  in  comparison  to  the  spectral  signature  plotted  in  Figure 
21.  Interestingly,  ^representation  has  revealed  the  movement  related  atom  similar  to  the  one 
already  observed  in  participant  A.  However,  now  the  coherence  pairs  showing  the  highest  loadings 
values  were  associated  with  P7  and  not  the  P8  electrode. 

5. 2. 3. 2. 2. 3  Participant  C 

The  PARAFAC  atomic  decomposition  revealed  three  distinct  atoms  related  to  workload,  task 
engagement/disengagement  and  the  atom  discriminating  the  counting  and  non-counting  periods. 
Although  this  was  already  observed  while  using  the  spectral  representation,  the  frequency 
distribution  of  the  Cxy  coherence  is  more  peaked  and  interestingly  the  vector  of  temporal  loadings 
indicate  no  mixing  among  the  periods  of  counting  and  the  relaxed  to  task  disengaged  periods.  This  is 
demonstrated  in  Figure  29. 
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Figure  29.  Loadings  of  the  ‘counting  versus  non-counting  related’  PARAFAC  atom  (Atom  3)  extracted 
from  EEG  recordings  of  Participant  C,  Day  2.  Top-Left  panel:  Spectral  signatures.  Top-right  panel:  Z- 
scores  of  the  temporal  signatures  of  the  EEG  atoms  numbered  in  the  right  panel.  Plotting  conventions  are 
the  same  as  in  Figure  26.  Bottom  panel:  The  coherence  plot  (connecting  lines)  of  the  electrode  pairs 
exceeding  the  90  percentile  of  the  all  coherence  values  distribution. 
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This  interesting  feature  of  the  coherence  representation  is  giving  raised  to  the  possibility  to  clearly 
discriminate  the  periods  of  task  engagement,  and  subsequently  to  discriminate  periods  of  counting 
versus  the  periods  of  non-counting.  The  spatial  distribution  of  high  loadings  (90  percentile)  shows 
the  importance  of  the  midline  oriented  parietal  and  occipital  coherence  pairs  at  the  frequency  peaked 
around  10.5  Hz.  Finally,  no  movement  related  atom  discriminating  the  periods  of  task  engagement 
from  relaxing  periods  was  observed  in  this  participant.  Finally,  no  movement  related  atom 
discriminating  the  periods  of  task  engagement  from  relaxing  periods  was  observed  in  this  participant. 

5. 2. 3. 2. 2. 4  Participant  D 

The  previously  reported  spectral  representation  results  indicated  the  periods  of  possible  electrodes  or 
measurement  device  failure  in  this  participant  during  the  first  day  experiment.  The  same  pattern  was 
observed  for  the  Cxy  coherence  measure.  In  contrast,  the  Pxy  and  Txy  coherence  measures  do  not  show 
these  extreme  temporal  loadings  drops  possibly  due  to  the  fact  that  the  values  of  these  measures  are 
normalized  using  the  power  spectrum  density  of  both  or  a  single  channel.  However,  none  of  these 
measures  have  shown  atoms  clearly  discriminating  the  different  workload  levels  or  engagement  vs. 
disengagement  periods.  This  is  not  surprising  because  the  noisy  element  in  the  data  cannot  be  simply 
filtered  out  just  by  using  the  different  coherence  measures.  Atomic  decomposition  similar  to  the  one 
already  depicted  in  Figure  25was  observed  when  using  the  Day  2  data. 

Participants:  a)  were  given  an  orientation  to  the  study,  b)  read  and  signed  an  informed  consent 
document,  c)  completed  a  brief  demographic  questionnaire  (age,  handedness,  hours  of  sleep,  etc.),  d) 
practiced  the  arithmetic  task  for  10  minutes,  and  e)  were  prepared  for  EEG  and  EOG  data  collection. 
They  then  completed  the  pretest  self-report  measures  (i.e.,  the  AD-ACL  and  VAMS)  and  performed 
the  mental  arithmetic  task  until  either  three  hours  had  elapsed  or  they  were  unwilling  to  continue. 
After  the  task,  they  completed  post-test  self-report  measures  and  were  debriefed. 

The  EEGs  were:  a)  submitted  to  an  algorithm  for  the  detection  and  elimination  of  eye-movement 
artifact,  b)  visually  examined  and  blocks  of  data  containing  artifact  were  manually  rejected,  c) 
epoched  around  the  stimulus  (i.e.,  from  -5  s  pre-stimulus  to  +8  s  post  -stimulus),  d)  low  pass  filtered 
(50  Hz;  zero  phase  shift;  12  dB/octave  roll  off),  and  e)  submitted  to  an  automated  artifact  rejection 
procedure  (i.e.,  absolute  voltages  >100  pV).  The  overall  single-epoch  rejection  rate  was  47%.  The 
‘cleaned  and  filtered’  epochs  were  decimated  to  a  sampling  rate  of  128  Hz.  EEG  power  spectra  were 
estimated  with  Welch’s  periodogram  method  at  833  frequencies  from  0-64  Hz. 

5.2.4  NASA-C  Database 

5.2.4.1  Methods 

5.2.4. 1. 1  Task  and  Data  Collection  Procedures 

Participants  sat  in  front  of  a  computer  with  their  right  hands  resting  on  a  4-button  keypad  and 
performed  arithmetic  summation  problems,  consisting  of  four  randomly  generated  single  digits,  three 
operators,  and  a  target  sum  (e.g.,  4+7-5+2=8),  which  were  displayed  on  a  computer  monitor 
continuously  until  the  subject  responded.  The  participants:  a)  solved  the  problems,  b)  decided 
whether  their  ‘calculated  sums’  were  less  than,  equal  to,  or  greater  than  the  target  sums  provided,  c) 
indicated  their  decisions  by  pressing  the  appropriate  key  on  the  keypad.  The  keypad  buttons  were 
labeled  “<,”  “=,”  and  “>,”  respectively.  Subjects  were  told  to  answer  as  quickly  as  possible  without 
sacrificing  accuracy.  After  a  response,  there  was  a  1  s  inter-trial  interval,  during  which  the  monitor 
was  blank.  Participants  performed  the  task  until  either  they  quit  from  exhaustion  or  3  h  had  elapsed. 
All  participants  performed  the  task  for  at  least  90  min  and  eleven  participants  completed  the 
maximum  3  h  period. 
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During  the  task,  the  EEG  was  recorded  continuously  using  32  Ag/AgCl  electrodes  embedded  in  a 
Quik-Cap™.36  The  reference  electrodes  were  averaged  mastoids  and  the  ground  electrode  was 
located  at  AFz.  Vertical  and  horizontal  EOG  were  recorded  using  bipolar  pairs  of  10  mm  Ag/AgCl 
electrodes  (i.e.,  one  pair  above  and  below  the  left  eye;  another  pair  to  the  right  and  to  the  left  of  the 
orbital  fossi).  Impedances  were  maintained  at  less  than  5  k£2  for  EEG  electrodes  and  less  than  10  k<T 
for  EOG  electrodes.  The  EEG  was  amplified  and  digitized  with  a  calibrated  64-channel  Synamps™ 
systemError!  Bookmark  no' def,ned-’  with  a  gain  of  1,000,  sampling  rate  of  500  Hz  and  a  pass  band  of  0.1  to 
100  Hz,  then  digitized  and  stored  on  magnetic  and  optical  media. 

5.2.4. 1.2  Spectral  representation  of  EEG 

In  the  first  step,  a  spectral  (PSD)  representation  of  the  EEG  was  used  in  the  PARAFAC  model.  First, 
the  PARAFAC  model  was  run  using  the  full  set  of  30  EEG  electrodes.  This  was  done  by  using  the 
data  from  each  participant  separately.  However,  as  in  the  other  databases  the  vector  of  electrode 
loadings  with  very  high  levels  indicated  a  noise  component  on  some  electrode  sites.  Very  high  noise 
levels  were  in  general  observed  at  the  temporal  sites  T7  and  T8  and  the  frontal  sites  Fpl,  Fp2,  Fz,  F4 
or  F5.  We  used  the  same  method  of  electrode  removal  as  for  the  APECS-W  algorithm  development 
described  in  the  preceding  sections.  The  electrodes  indicating  high  level  of  noise  were  selectively 
removed  from  each  participant’s  data  and  the  PARAFAC  model  was  run  again.  This  was  done  for 
each  participant  separately.  In  all  cases  this  resulted  into  the  extraction  of  the  physiologically  more 
plausible  results  in  comparison  to  the  models  with  the  full  EEG  electrodes  set.  In  addition  we  ran  the 
PARAFAC  model  twice  for  each  participant.  After  the  convergence  of  the  first  run  the  points  which 
exceeded  the  90th  percentile  (lower  than  the  95  or  99  percentiles  used  for  APECS-W)  of  the  residual 
variance  and  leverage  distributions  (i.e.,  points  indicating  noisy  samples)  were  removed  and  the 
PARAFAC  model  was  run  again.  In  general,  this  procedure  removed  points  with  very  high  values  of 
temporal  loadings  (signatures).  The  core  consistency10  of  these  final  models  was  in  all  cases  greater 
than  80%  indicating  a  good  model  fits. 

We  experimented  with  two-atom,  three-atom,  and  four-atom  models.  Across  subjects,  the  three  atom 
models  were  often  sufficient  to  explain  all  of  the  experiment-wise  variance.  For  most  subjects, 
however,  four-atom  models  provided  fits  that  were  more  plausible  physiologically.  In  a  few  subjects, 
four-atom  models  resulted  in  factors  that  appeared  to  split  variance  spatially  but  preserve  variance 
temporally  and  spectrally.  For  this  reason  we  limited  our  further  investigations  to  three-  or  four-atom 
models  choosing  the  most  appropriate  model  for  each  subject.  Unlike  the  APCES-W  algorithms  and 
the  coherence  analyses  presented  below,  for  all  atomic  models  in  these  spectral  analyses,  we  did  not 
constrain  the  solutions  for  loadings  to  be  nonnegative  or  unimodal,  i.e.,  no  constraints.  These  PSD 
analyses  were  done  first,  and  were  extensive,  and  only  later  we  appreciated  that  constraints  would 
lead  to  more  easily  interpretable  models,  especially  for  PSD  and  coherence  measures.  However,  we 
chose  to  leave  the  PSD  analyses  in  the  unconstrained  state  to  allow  time  for  the  coherence  models, 
which  were  more  interesting  and  novel. 

We  present  results  from  a  two-atom  model  fit  to  one  participant’s  data  to  illustrate  the  effects  of  a 
noisy  electrode  (FI).  For  this  demonstration,  a  two-atom  model  was  fit  to  the  data.  The  first  atom 
(blue  line)  had  an  extremely  large  loading  on  a  single,  noisy  electrode  (F7).  This  atom  had  a  broad 
spectrum,  and  loadings  on  other  electrodes  were  relatively  small  and  unstructured.  In  contrast,  a 
second  atom  had  a  peak  near  9  Hz  in  the  power  spectrum  and  loadings  that  were  selectively  high  at 
central  electrode  sites,  especially  Pz  and  CPz.  This  suggests  a  close  association  of  this  atom  with  the 
alpha  rhythm  of  the  EEG.  Interestingly,  the  loadings  of  the  second  atom  in  the  time  dimension 
clearly  reflect  the  development  of  fatigue  between  the  first  15  minutes  of  task  performance  (alert 
period)  and  the  final  15  minutes  (fatigued  period).  This  finding  is  consistent  with  effects  of  increased 
power  in  the  alpha  band  during  the  development  of  mental  fatigue  we  observed  previously  with  this 
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data  setError!  Bookmark  not  defined.  Qn  other  hand  the  time  loadings  of  Atom  1,  with  the  broad 
spectrum  and  noisy  electrode  do  not  reflect  the  development  of  fatigue.  This  suggests  a  potential 
method  based  on  atomic  decomposition  for  removal  of  contaminants  from  data  that  are  uncorrelated 
with  the  experimental  treatments  or  effects. 
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Figure  30.  A  two-atom  model  fit  to  EEG  from  one  participant  GSD  illustrates  the  effects  of  a  noisy 
electrode  (FI).  Top  left  panel:  the  Atom  1  (blue  line)  had  an  extremely  large  loading  on  a  single,  noisy 
electrode  (F7),  where  as  the  Atom  2  (red  line)  had  selectively  high  loadings  at  central  electrode  sites, 
especially  Pz  and  CPz.  Top  right  panel:  Atom  1  (blue  line)  had  a  broad  spectmm,  whereas  Atom  2  (red 
line)  had  a  single  peak  near  9  Hz  in  the  power  spectrum.  Lower  panel:  the  loadings  of  Atom  2  (red  line)  in 
the  time  dimension  clearly  reflect  the  development  of  fatigue  between  the  first  1 5  minutes  of  task 
performance  (alert  period)  and  the  final  15  minutes  (fatigued  period)  unlike  Atom  1  (blue  line).  Time  is 
measured  in  epochs  *  seconds  and  is  not  continuous  due  to  gaps  where  EEG  artifacts  where  rejected.  A 
smoothing  filter  with  a  span  of  0.10  was  used  to  illustrate  trends. 

5. 2. 4. 1.3  General  observations 

We  performed  an  analysis  of  the  PSDs  from  EEG  recordings  of  12  participants  in  the  NASA-C  study. 
The  time  on  task  varied  considerably  among  participants,  ranging  from  less  than  one  hour  to  three 
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hours.  Prior  results  had  shown  clearly  that  fatigue  developed  during  the  first  15  to  45  minutes  in  most 
participants,  so  we  expected  to  observe  a  contrast  between  alert  and  fatigued  periods  when 
participants  performed  the  task  for  at  least  one  hour.  Participants  who  did  not  complete  at  least  one 
hour  of  task  performance  were  excluded  from  the  analyses.  The  analyses  are  extensive  and  will 
require  additional  examination  and  statistics  for  general  publication.  Nevertheless  the  results  are 
striking  and  clearly  illustrate  the  power  of  the  atomic  decomposition  method  for  EEG-based 
estimation  of  mental  fatigue.  Eleven  of  the  12  participants  showed  evidence  of  EEG  atoms  in  the 
alpha  and  theta  bands  that  changed  significantly  over  time  on  task.  Each  subject’s  results  are 
summarized  in  a  three-part  figure,  similar  to  Figure  30.  Because  the  figures  are  large  and  numerous, 
we  have  included  them  in  an  appendix  for  ease  of  reference  (Appendix  31. 

5.2.4. 1.4  Three  classes  of  participants 

The  figures  in  Appendix  3  contain  vast  amounts  of  information  that  will  require  additional  analyses. 
However  to  the  point  of  this  study,  we  are  able  to  confirm  the  effectiveness  of  atomic  decomposition 
for  isolating  atoms  that  are  sensitive  to  the  development  of  mental  fatigue.  As  the  figures  in 
Appendix  3  show,  we  gauged  the  significance  of  a  change  in  the  loadings  of  an  atom  over  time  on 
task  by  measuring  each  series  of  loadings  as  a  set  of  standard  scores  (z-scores)  which  were 
normalized  to  the  mean  and  variance  of  the  first  fifteen-minute  block,  when  all  subjects  were  alert. 

We  are  also  able  to  distinguish  individual  differences  that  roughly  sort  the  participants  into  three 
classes.  The  first  class  contains  the  vast  majority  of  participants  and  is  characterized  by  having  atoms 
that  reflect  EEG  power  in  the  theta  and  alpha  bands,  where  the  loadings  in  both  bands  change 
significantly  over  the  course  of  the  task.  This  class  includes  eight  of  the  12  participants:  ARB,  GSD, 
JCH,  KTT,  MMB,  MMC,  RWC,  and  RGR.  The  second  class  consisted  of  participants  who  had  only 
one  atom  that  changed  significantly  over  time  on  task.  This  class  included  JCS,  who  had  only  a  theta- 
related  atom  sensitive  to  fatigue,  and  SKH  and  TBN,  who  had  only  an  alpha-related  atom  sensitive  to 
fatigue.  The  third  class  included  a  single  subject,  WXS,  whose  EEG  atoms  did  not  clearly  reflect  a 
trend  over  time  on  task.  In  some  cases  changes  in  loadings  of  atoms  over  time  were  increases  (mostly) 
and  in  other  cases  they  were  decreases.  The  meaning  of  the  direction  of  change  in  an  atom’s  loadings 
is  not  readily  interpretable,  as  the  loadings  in  other  dimensions  can  be  negative  or  positive,  and  the 
sense  of  the  loadings  can  be  inverted  without  changing  the  solution  of  the  model  from  the  data. 
Separate  analyses  will  be  required  to  interpret  the  meaning  of  directions  of  change  in  the  loadings. 

5.2.4. 1.5  Coherence  representation  of  EEG 

As  for  the  other  two  databases,  we  used  three  types  of  the  coherence  estimates:  1)  the  Cross  Power 
Spectral  Density  (CPSD)  estimate  via  Welch's  method  (here  denoted  Cxy),  2)  the  Magnitude  Squared 
Coherence  Estimate  (denoted  Pxy),  3)  the  estimate  of  the  transfer  function  of  the  system  using  the 
Welch's  averaged  periodogram  method  (denoted  Txy). 

As  for  the  spectral  representations,  we  experimented  with  two-component,  three-component,  and 
four-component  models.  Again,  across  subjects,  the  four  component  models  were  sufficient  to 
explain  all  of  the  experiment-wise  variance  and  for  coherence,  we  limited  our  further  investigations 
to  four-component  models  only.  For  individual  participants  the  same  EEG  electrodes  were  used  as  in 
the  case  of  spectral  representation.  Coherence  among  all  possible  electrode  pairs  was  computed  and 
used  in  the  model,  excluding  self-pairings  of  electrodes. 

5. 2. 4. 1.6  General  observations 

As  for  the  PSDs,  we  performed  an  analysis  of  the  magnitude-squared  coherences,  or  Cxy,  of  EEG 
recordings  from  12  participants  in  the  NASA-C  study.  Prior  results  with  coherence  measures  had  not 
been  examined  for  these  data,  so  we  followed  the  analysis  approach  used  for  the  PSDs.  Again, 
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participants  who  did  not  complete  at  least  one  hour  of  task  performance  were  excluded  from  the 
analyses.  Fike  the  PSD  analyses,  the  coherence  analyses  are  also  extensive  and  will  require 
additional  examination  and  statistics  for  general  publication.  Each  subject’s  results  are  summarized 
in  a  three-part  figure,  similar  to  Figure  26,  but  with  some  important  differences.  Because  the  number 
of  electrodes  was  considerably  higher  than  for  the  APECS-W  algorithm  developments,  we  changed 
the  topographical  representation.  We  use  instead  a  mapping  algorithm  that  projects  the  spherical 
coordinates  of  the  EEG  electrodes  on  a  plane,  while  preserving  the  sense  of  distance  and  curvature  of 
the  head.  This  is  known  in  map  projection  geometry  as  the  Werner  projection,  a  nonconformal 
equal-area  projection.37  Although  this  projection  is  seldom  used  in  modem  cartography,  we  found  it 
useful  for  mapping  EEG  coherences  because  the  projection  helps  to  minimize  overlap  of  lines 
connecting  pairs  of  electrodes. 


Figure  31.  Spherical-planar  projection  used  to  map  electrode  locations  and  links  representing  coherences. 
Left  panel,  in  the  Werner  Projection ;  sectors  delimited  by  standard  meridians  and  parallels  have  equal 
area  and  no  overlap.  Right  panel,  our  projection  for  mapping  EEG  coherences,  where  the  North  Pole 
(center)  represents  the  vertex,  Cz,  frontal  direction  is  up,  occipital  direction  is  down,  and  the  right  and  left 
are  mapped  as  East  and  West.  The  origin  of  our  projection  is  70°N,  180°W.  The  standard  meridians  lie  at 
increments  of  18°  and  standard  parallels  lie  at  increments  30°,  45°,  and  67.5°,  which  align  approximately 
with  lines  of  the  International  10-20  system  of  EEG  electrode  placement. 

As  for  the  PSD  analyses,  the  figures  for  the  coherence  analyses  are  large  and  numerous,  so  we  have 
also  included  them  in  an  appendix  for  ease  of  reference  (Appendix  4).  The  results  are  complex,  but 
also  quite  striking.  They  reveal  a  complexity  of  interconnections  and  marked  individual  differences 
in  the  EEG  correlates  of  mental  fatigue.  Nevertheless  there  are  some  unmistakable  commonalities 
among  the  coherence  signatures  of  certain  atoms  among  the  participants.  Whereas  11  of  the  12 
participants  showed  evidence  of  PSD  atoms  sensitive  to  mental  fatigue,  all  of  the  participants  showed 
evidence  of  coherence  atoms  sensitive  to  mental  fatigue. 

Obviously  this  is  a  rich  and  interesting  set  of  results,  which  call  for  deeper  analysis  than  afforded  by 
this  limited  research  program.  Here,  we  begin  this  analysis  by  summarizing  the  results  according  to 
these  broad  parameters  (Table  2).  The  patterns  are  involved,  but  we  can  summarize  them  in  terms  of 
three  broad  parameters:  coherence  spectrum  (theta  effects,  alpha  effects,  and  effects  in  non-specific 
bands  such  as  very  low-frequency,  high-frequency,  or  broadband  effects),  spatial  focus  (frontal 
central,  parietal),  and  range  of  coherence  (short,  medium,  long).  It  is  clear  that  all  regions  of  the  EEG 
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scalp  distribution  are  affected  by  fatigue  to  some  degree,  in  some  cases  with  a  narrow  focus,  and  in 
others  with  long-range  and  diverse  connections. 

The  summary  shows  that  11  of  12  participants  had  evidence  of  an  atom  with  a  focus  in  the  theta  band. 
Of  these  11,  eight  had  medium  to  long-range  coherence  interactions  with  central  and  parietal  regions. 
In  seven  of  the  12  participants,  there  was  evidence  of  an  atom  with  a  focus  in  the  alpha  band.  Of 
these  seven,  five  had  medium  to  long-range  interactions  with  central  or  frontal  regions.  Four 
participants  also  showed  evidence  of  nonspecific  or  broadband  effects,  all  of  these  with  medium  to 
long-range  interactions  and  mostly  in  frontocentral  regions.  The  one  participant  who  did  not  have  a 
clear  effect  of  fatigue  on  atoms  derived  from  PSDs  had  two  nonspecific  atoms  which  were  clearly 
sensitive  to  mental  fatigue,  one  for  low-frequency  EEG  coherences,  and  another  for  high-frequency 
coherences. 


Table  2.  Summary  of  EEG  coherence  atoms  among  the  12  participants  of  the  NASA-C  study. 


Participant 

Theta  Effect 

Alpha  Effect 

Other/broadband  Effect 

Spatial  Focus 

Range 

Spatial  Focus 

Range 

Spatial  Focus 

Range 

ARB 

Frontal 

Med-long 

Parietal 

Med-long 

GSD 

Central 

Med-long 

Parietal 

Med-long 

JCH 

Frontal 

Med-long 

JCS 

Parietal 

Medium 

Frontocentral 

Med-long 

Frontoparietal 

Long 

KTT 

Frontocentral 

Med-long 

Frontal 

Med-long 

MMB 

Frontal 

Short-med 

Frontal 

Med-long 

Central 

Med-long 

MMC 

Centroparietal 

Medium 

Parietal 

Med-long 

RGR 

Frontocentral 

Medium-long 

Centroparietal 

Short-med 

RWC 

Central 

Short-med 

Frontal 

Short-med 

Parietal 

Short-med 

SKH 

Frontocentral 

Med-long 

TBN 

Frontoparietal 

Long 

WXS 

Frontocentral 

Long 

Frontocentral  (lo-F) 

Med-long 

Frontocentral  (hi-F) 

Med-long 

5.2.4.2  Discussion 

As  for  the  APEC-W  algorithms  applied  to  mental  workload  estimation  in  the  USAF-C  and  UST-T 
databases,  the  APECS-F  algorithms  provided  atomic  decompositions  that  isolated  frequency  bands, 
electrode  locations,  and  coherent  interactions  that  were  significantly  associated  with  the  development 
of  mental  fatigue.  Of  the  two  approaches,  PSD  and  coherence,  the  latter  seemed  to  provide  a  more 
thorough  and  precise  description  of  fatigue  effects  across  all  participants.  In  particular,  it  is  evident 
that  mental  fatigue  generally  involves  changes  in  coherences  in  the  theta  band,  which  are  distributed 
in  the  frontocentral  and  central  regions  and  involve  medium-  to  long-range  interactions.  Surprisingly, 
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short-range  coherences,  although  present  in  atoms  of  most  participants,  were  not  often  associated 
with  mental  fatigue,  as  gauged  by  time  on  the  mental  arithmetic  task.  This  result  reinforces  the  notion 
that  the  coherences  linked  to  fatigue-sensitive  atoms  are  not  artifacts  of  the  method  because  it  is  well 
known  that  the  highest  coherences  generally  occur  between  closely  spaced  electrodes  and  tend  to  fall 
of  as  the  inter-electrode  distance  increases. 

6  General  discussion  and  conclusions 

This  project  has  proven  to  be  an  extremely  fruitful  and  penetrating  view  of  an  entirely  new  approach 
to  multidimensional  analysis  of  experiments  in  which  EEG  is  used  to  detect  changes  in  mental  states. 
The  results  speak  conclusively  to  the  fact  the  atomic  decomposition  provides  a  novel  view  and 
powerful  insight  concerning  the  interactions  of  brain  regions  and  oscillatory  EEG  sources  as  they 
change  with  mental  states.  Although  the  results  are  impressive,  we  feel  that  this  project  has  barely 
scratched  the  surface  of  the  potential  for  application  of  atomic  decomposition  to  EEG. 

7  Future  research  directions 

Of  most  importance  to  the  US  Army  and  its  need  to  accurately  assess  operator  functional  states,  the 
methods  we  have  developed  here  should  be  extended  in  two  important  directions.  First,  as  with  our 
prior  work  using  PLS  and  linear  or  nonlinear  classifiers,  we  must  use  atomic  decomposition  to 
extract  features  of  EEG  that  serve  as  inputs  to  classifiers  of  mental  states.  Our  experience  with  such 
classification  work  and  testing  will  allow  us  to  rapidly  develop  these  methods  and  apply  them  to  the 
existing  data  sets  with  minimal  effort.  Secondly,  as  atomic  decomposition  is  relatively  new  in  the 
analysis  of  EEG,  we  must  design  specific  experiments  that  will  allow  us  to  test  hypotheses 
concerning  the  validity  and  utility  of  the  method  in  controlled  studies.  For  this  to  succeed  we  will 
seek  partnerships  with  experimental  groups  and  add  our  methodology  to  ongoing  and  planned 
experiments,  to  be  as  efficient  as  possible  in  the  early  development  of  the  methodology.  Should 
atomic  decomposition  methods  prove  generally  valid  and  useful  in  EEG  applications,  we  will  aim  to 
publish  the  fundamental  advances  in  academic  journals  and  distribute  the  underlying  software 
technology  through  commercial  avenues. 
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Abstract:  This  paper  describes  the  use  of  multi-way  decomposition  methods  to  efficiently  summarize 
electroencephalographic  (EEG)  data.  A  space-frequency-time  atomic  decomposition  was  applied  to  EEG 
data  recorded  while  subjects  performed  tasks  associated  with  varying  levels  of  cognitive  performance.  The 
new  atomic  decomposition  of  cognitive  workload  data  revealed  alpha  and  theta  EEG  oscillations  which 
agree  with  observations  reported  in  the  brain  research  literature.  The  temporal  signature  of  the  atoms 
discriminates  between  different  levels  of  cognitive  activity.  The  results  and  analysis  confirm  the  utility  of 
the  multi-way  decomposition  method  to  construct  new  models  and  algorithms  for  monitoring  cognitive 
status,  which  can  supplement  or  overcome  existing  approaches  based  on  conventional  two-dimensional 
space-time  or  frequency-time  data  decomposition. 

Keywords:  multi-way  decomposition,  PARAFAC,  cognitive  workload 

1  Introduction 

Accurate  and  timely  estimation  and  classification  of  operator  functional  state  has  important  practical 
consequences  in  many  safety- critical  operational  situations.  Impairment  of  cognitive  performance  in  subjects  during 
a  safety- critical  task  poses  a  high  risk  for  procedural  errors,  often  leading  to  severe  consequences  with  high  economic 
or  life  losses.  Cognitive  impairment  has  been  documented  in  different  operational  situations,  and  attributed  to  several 
factors,  including  time  on  duty,  sleep  loss,  extended  time  on  single  tasks,  unusually  high  mental  workload, 
psychosocial  stress,  exposure  to  neurotoxins  and  vestibular  dysfunction. 

Despite  the  extensive  prior  efforts  to  develop  physiological  methods  for  monitoring  arousal  and  cognition, 
methods  for  monitoring  mental  workload  are  still  unreliable.  EEGs  are  considered  to  be  the  gold  standard  for 
objective  detection  of  mental  status  and  cognitive  function  and  currently  may  be  recorded  and  analyzed  in  almost  any 
occupational  setting.  However,  EEGs  are  usually  recorded  as  a  high- dimensional  in  time- space  distributed  data  and 
many  conventional  2-D  decomposition  techniques  (for  example,  principal  component  analysis  (PC A)  or  independent 
component  analysis,  Hyvarinen  et  al.,  2001)  may  not  be  ideally  suited  to  reveal  the  existing  latent  data  structure, 
because  unfolding  2-D  decompositions  can  be  done  in  several  ways,  and  interactions  of  dimensions  are  not  modeled 
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well  in  2-D  decompositions.  Recently,  promising  results  were  achieved  by  applying  multi-way  decomposition 
models  such  as  PARAFAC  or  TUCKER3  to  EEG  data  (Estienne  et  al.,  2001;  Miwakeichi  et  al.,  2004).  Using  these 
multi-way  decomposition  models  we  analyzed  EEG  data  recorded  from  participants  in  a  study  of  mental  workload. 
To  our  knowledge  this  is  the  first  time  that  multi-way  decomposition  models  have  been  applied  to  the  problem  of 
mental  workload  analysis  and  monitoring. 

2  Methods 

2.1  Data  acquisition  and  preprocessing 

Two  subjects  were  trained  to  stable  performance  on  a  simulated  Unmanned  Air  Vehicle  (UAV)  task.  The  task 
consists  of  monitoring  of  the  progress  of  four  UAVs  as  they  flew  a  preplanned  mission.  Each  subject  participated  in 
three  in  time-separated  sessions.  Five  channels  of  EEG  (electrodes  located  at  Fz,  F7,  Pz,  T5  and  02  with  a  linked 
mastoid  reference),  ECQ  vertical  and  horizontal  EOG  were  recorded.  In  this  study  only  the  EEG  data  were  used.  The 
EEG  data  were  sampled  at  200  Hz  and  low-pass  filtered  with  cutoff  of  30  Hz.  Next,  the  data  were  segmented  into  2-s 
long  windows  with  an  overlap  of  500  ms.  For  each  segment  the  power  spectral  density  (PSD)  was  computed  using 
the  Thomson  multitaper  method  (Thomson,  1982).  Frequencies  in  the  range  of  2.5  to  20  Hz  were  considered  in  this 
study.  Initial  analyses  revealed  a  high  level  of  power  at  frequencies  below  4  Hz.  Power  at  these  frequencies  often 
arose  from  motion  artifacts  and  confounded  the  PARAFAC  analyses  of  EEG.  For  this  reason  only  frequencies  from  5 
to  20  Hz  were  considered  for  further  analyses.  This  procedure  was  repeated  for  each  EEG  channel  separately  and  a 
three-dimensional  matrix  X  @  xj  xk)  with  I  time  segments,  J  electrodes  and  PSD  estimates  at  K  frequencies  was 
constructed. 

2.2  The  PARAFAC  model 

A  three-way  PARAFAC  model  was  applied  to  data.  PARAFAC  can  be  seen  as  a  generalization  of  PC  A  for 
dealing  with  multi-dimensional  data  (Bro,  1997).  Three  loading  matrices,  A,  B,  and  C  with  elements  a ^  b $  and  ckf 
defines  the  model  that  can  be  mathematically  described  as 

F 

xijk  =  2  aV^]fckf  +  Sijk 

f= 1 

where  are  the  residual  elements  or  errors  and  F  stands  for  number  of  components  or  atoms  that  are 
considered.  The  loadings  elements  are  then  found  by  minimizing  the  sum  of  squares  of  the  residuals  syu  (Bro,  1997), 
that  is 

F 

min  ||^-ZVV<VH2 

akrjsfckf  f=i 

For  the  analyses  reported  here  we  used  proprietary  m-codes  developed  by  Pacific  Development  and  Technology, 
LLC,  and  subroutines  from  the  A7- way  toolbox  for  Matlab®  (Anders son  and  Bro,  2000). 

3  Results 

A  three-atom  PARAFAC  model  was  observed  to  provide  acceptable  and  physiologically  interpretable  results  for 
EEG  recordings  of  both  subjects,  for  which  core  consistency  diagnostic  values,  which  index  the  stability  of  the 
decompositions,  reached  87%  and  97%,  respectively  (Bro  and  Kiers,  2003;  Andersson  and  Bro,  2000). 

Three  distinct  spectral  signature  atoms  (components)  were  observed  in  the  first  subject  (Figure  1,  right).  Atom  1 
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any  specific  rhythm  or  band,  and  most  likely  reflects  background  EEG  Atom  2  shows  a  concentration  of  high 
loadings  for  frequencies  in  the  theta  band.  Loadings  for  Atom  3  were  maximal  in  the  band  5-9.5  Hz.  The  spatial 
loadings  of  Atom  1  were  the  same  as  for  Subject  1  (Fz,  Pz,  Oz),  whereas  those  of  Atom  2  were  maximal  at  Fz, 
indicating  an  association  with  theta  rhythm,  The  spatial  loadings  of  Atom  3  were  maximal  at  Fz  and  Pz,  also 
indicating  association  with  theta  and  low'  alpha  rhythms.  The  temporal  loadings  of  Atoms  1  and  2  did  not  show  a 
marked  relationship  to  workload  conditions;  however,  the  temporal  loadings  of  Atom  3  tracked  closely  with  the 
various  mental  workload  conditions  and  rest  periods. 


Atom  1 


Atom  2 


Atom  S 


Atom  3 

Enlarged 

Scale 


Figure  2  Subject  2.  Left  panel:  Temporal  signatures  of  the  BEG  decomposition.  Light  gray  areas  mark  lower  mental  workload  states 
while  the  higher  workload  states  are  marked  by  the  dark  gray  areas.  Vertical  dotted  lines  separate  three  in  time  distinct  experimental 
sessions.  Bottom  panel:  The  third  temporal  atom  alone.  Right  panel:  Spectral  signatures  corresponding  to  temporal 

atoms  plotted  in  the  left  panel. 


PARAFAC  Loadings 


Time  (m) 


4  Conclusions 

The  results  of  this  study  show  that  mental  workload  may  be  tracked  by  LEG  components  isolated  using  parallel 
factor  analysis  or  PARAFAC.  Components  or  atoms  of  the  PARAFAC  decomposition  bad  high  loadings  in  the 
frequency  spectrum  and  across  electrodes,  which  reflected  alpha,  theta,  and  broad- band  REG  processes.  Unlike  other 
approaches  to  isolate  EEG  factors  related  to  mental  workload,  our  application  of  PARAFAC  begins  with  a  truly 
three-dimensional  model  of  EEG  variance  observed  during  performance  of  a  cognitive  task.  This  model  is  fit  to  the 
EEG  simultaneously  in  frequency,  space,  and  time.  The  time  dimension  in  the  present  study  reflects  imposed  task 
demands  that  produced  calibrated  states  of  low-or  high  mental  workload. 

For  an  unsupervised  method  of  decomposition,  these  PARAFAC  results  are  remarkable.  Using  a  small  number 
of  electrodes,  the  loadings  of  several  PARAFAC  atoms  in  time,  co- varied  with  task  demands  and  mental  workload. 
Admittedly,  these  results  are  based  on  a  small  sample  of  two  subjects.  However,  the  task  performances  were  long, 
providing  about  30  minutes  of  EEG  recordings  for  analysis.  In  our  extensive  experience  with  similar  experiments, 
the  discovery  of  EEG  atoms  that  track  workload  and  have  meaningful  spectral  and  spatial  propertied  from  recordings 
of  these  durations  is  not  likely  to  arise  from  chance. 
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Part  2  Application  Research 

We  are  extending  our  research  in  two  directions.  First  we  are  increasing  the  number  of  tasks  and  subjects  in 
which  we  are  applying  the  PARAFAC  method.  Two  additional  studies  are  in  process,  including  a  cognitive  fatigue 
study  and  a  warfare  simulation  task.  Second,  we  are  now  testing  a  fourth  dimension  in  the  PARAFAC  model: 
coherence  of  EEG  sources.  Short-and  long  range  coherence  of  alpha  and  theta  sources  are  expected  to  increase  with 
cognitive  effort,  based  on  theoretical  studies  of  EEG  dynamics  and  considerable  experimental  data  (Nunez  and 
Srinivasan,  2005).  We  expect  models  that  can  encompass  the  direction,  bandwidth,  and  extent  of  EEG  coherence  will 
help  to  further  define  neural  sources  that  interact  in  cognition. 
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PROTOCOL  SHEET 

Gamer  Study  No.  2 


TASK 

TONES* 

COUNT 

3.5  minutes  with  1  enemy 

NO 

NO 

□ 

3.5  minutes  with  0  enemies 

NO 

NO 

□ 

7  minutes  with  1  enemy 

NO 

NO 

□ 

7  minutes  with  1  enemy 

YES 

YES 

□ 

7  minutes  with  5  enemies 

NO 

NO 

□ 

7  minutes  with  5  enemies 

YES 

YES 

Alcohol  or  placebo  administration  occurs  here. 

□ 

3.5  minutes  with  0  enemies 

NO 

NO 

□ 

7  minutes  with  1  enemy 

NO 

NO 

□ 

7  minutes  with  1  enemy 

YES 

YES 

□ 

7  minutes  with  5  enemies 

NO 

NO 

□ 

7  minutes  with  5  enemies 

YES 

YES 

□ 

3.5  minutes  with  0  enemies 

NO 

NO 

□ 

3.5  minutes  with  0  enemies 

YES 

YES 

*“No”  means  that  the  participant  does  not  need  to  listen  to  tones  and  does  not  report  kills  or  death 
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Figures  for  PSD  Analyses  of  12  Participants’  EEG  from  the  NASA-C  Database 


Subject:  ARB  Epoch:  2  s  Overlap:  0  s  Method:  Welch 


Figure  32.  Atomic  decomposition  of  EEG  from  participant  ARB  of  the  NASA-C  study.  Participants 
solved  mental  arithmetic  problems  on  a  computer  for  up  to  three  hours  without  rest.  EEG  recordings  from 
30  channels  were  processed  using  PARAFAC  decomposition  to  yield  a  model  consisting  of  four  atoms, 
each  have  dimensions  of  space  (electrodes),  frequency  (power  spectral  density)  and  time  (time  on  task). 
All  panels :  Atoms  1-4  are  represented  by  colored  lines  in  the  order  blue,  red,  green,  cyan.  Upper  left 
panel:  distributions  of  loadings  for  EEG  atoms  across  a  32-channel  montage.  Certain  electrodes  may  have 
been  excluded  from  analyses  if  they  were  judged  to  contain  high  levels  of  noise,  as  illustrated  in  Figure 
30.  In  this  participant,  electrodes  F7  and  FP1  were  excluded.  Upper  right  panel :  distribution  of  loadings 
for  EEG  atoms  across  frequency  bins  analyzed  in  the  PSD  estimate.  Lower  panel :  distributions  of 
loadings  for  EEG  atoms  across  time  on  task.  The  fine  vertical  lines  separate  15-minute  blocks  of  time. 

The  first  two  blocks  are  consecutive  and  were  at  the  beginning  of  the  session.  The  last  two  blocks  were 
also  consecutive  and  were  at  the  end  of  the  session.  This  participant  performed  the  task  for  one  hour,  or 
four  15-minute  blocks.  The  time  axis  measures  time  as  multiples  of  2-s  long  EEG  epochs  which  were  not 
all  contiguous,  due  to  rejection  of  EEG  segments  containing  movement  or  other  artifacts.  The  last  pair  of 
blocks  has  fewer  epochs  because  the  incidence  of  EEG  artifacts  increased  over  time  in  this  participant. 
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Figure  33.  Atomic  decomposition  of  EEG  from  participant  GSD  of  the  NASA-C  study.  EEG  recordings 
from  30  channels  were  processed  using  PARAFAC  decomposition  to  yield  a  model  consisting  of  four 
atoms,  each  have  dimensions  of  space  (electrodes),  frequency  (power  spectral  density)  and  time  (time  on 
task).  Graphical  conventions  are  the  same  as  in  Figure  32.  This  participant  performed  the  task  for  three 
hours,  or  12  15-minute  blocks.  The  time  axis  measures  seconds  as  multiples  of  2-second  long  EEG 
epochs  which  were  not  all  contiguous,  due  to  rejection  of  EEG  segments  containing  movement  or  other 
artifacts.  Some  blocks  have  fewer  epochs  than  others  because  the  incidence  of  EEG  artifacts  increased 
during  those  blocks. 


Page  A3 -2 


APECS  Final  Report 
September  15,  2009 


ARO  Contract  No.  W91 1NF08CO121 
PDT  Report  No.  UA01BF0636B131 


Subject:  JCH  Epoch:  2  s  Overlap:  0  s  Method:  Welch 


1200  2400  3600  4800  6000  7200  8400  9600 


Time(s) 


Figure  34.  Atomic  decomposition  of  EEG  from  participant  JCH  of  the  NASA-C  study.  EEG  recordings 
from  30  channels  were  processed  using  PARAFAC  decomposition  to  yield  a  model  consisting  of  four 
atoms,  each  have  dimensions  of  space  (electrodes),  frequency  (power  spectral  density)  and  time  (time  on 
task).  Graphical  conventions  are  the  same  as  in  Figure  32.  This  participant  performed  the  task  for  three 
hours,  or  12  15-minute  blocks.  The  time  axis  measures  seconds  as  multiples  of  2-second  long  EEG 
epochs  which  were  not  all  contiguous,  due  to  rejection  of  EEG  segments  containing  movement  or  other 
artifacts.  Some  blocks  have  fewer  epochs  than  others  because  the  incidence  of  EEG  artifacts  increased 
during  those  blocks. 
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Figure  35.Atomic  decomposition  of  EEG  from  participant  JCS  of  the  NASA-C  study.  EEG  recordings 
from  30  channels  were  processed  using  PARAFAC  decomposition  to  yield  a  model  consisting  of  four 
atoms,  each  have  dimensions  of  space  (electrodes),  frequency  (power  spectral  density)  and  time  (time  on 
task).  Graphical  conventions  are  the  same  as  in  Figure  32.  This  participant  performed  the  task  for  1.75 
hours,  or  seven  15-minute  blocks.  The  time  axis  measures  seconds  as  multiples  of  2-second  long  EEG 
epochs  which  were  not  all  contiguous,  due  to  rejection  of  EEG  segments  containing  movement  or  other 
artifacts.  Some  blocks  have  fewer  epochs  than  others  because  the  incidence  of  EEG  artifacts  increased 
during  those  blocks. 
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Figure  36.  Atomic  decomposition  of  EEG  from  participant  KTT  of  the  NASA-C  study.  EEG  recordings 
from  30  channels  were  processed  using  PARAFAC  decomposition  to  yield  a  model  consisting  of  four 
atoms,  each  have  dimensions  of  space  (electrodes),  frequency  (power  spectral  density)  and  time  (time  on 
task).  Graphical  conventions  are  the  same  as  in  Figure  32.  This  participant  performed  the  task  for  three 
hours,  or  12  15-minute  blocks.  The  time  axis  measures  seconds  as  multiples  of  2-second  long  EEG 
epochs  which  were  not  all  contiguous,  due  to  rejection  of  EEG  segments  containing  movement  or  other 
artifacts.  Some  blocks  have  fewer  epochs  than  others  because  the  incidence  of  EEG  artifacts  increased 
during  those  blocks. 


Page  A3 -5 


APECS  Final  Report 
September  15,  2009 


ARO  Contract  No.  W91 1NF08CO121 
PDT  Report  No.  UA01BF0636B131 


Subject:  MMB  Epoch:  2  s  Overlap:  0  s  Method:  Welch 


Figure  37.  Atomic  decomposition  of  EEG  from  participant  MMB  of  the  NASA-C  study.  EEG  recordings 
from  30  channels  were  processed  using  PARAFAC  decomposition  to  yield  a  model  consisting  of  four 
atoms,  each  have  dimensions  of  space  (electrodes),  frequency  (power  spectral  density)  and  time  (time  on 
task).  Graphical  conventions  are  the  same  as  in  Figure  32.  This  participant  performed  the  task  for  2.5 
hours,  or  10  15-minute  blocks.  The  time  axis  measures  seconds  as  multiples  of  2-second  long  EEG 
epochs  which  were  not  all  contiguous,  due  to  rejection  of  EEG  segments  containing  movement  or  other 
artifacts.  Some  blocks  have  fewer  epochs  than  others  because  the  incidence  of  EEG  artifacts  increased 
during  those  blocks. 
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Figure  3  8.  Atomic  decomposition  of  EEG  from  participant  MMC  of  the  NASA-C  study.  EEG  recordings 
from  30  channels  were  processed  using  PARAFAC  decomposition  to  yield  a  model  consisting  of  three 
atoms,  each  have  dimensions  of  space  (electrodes),  frequency  (power  spectral  density)  and  time  (time  on 
task).  Graphical  conventions  are  the  same  as  in  Figure  32.  This  participant  performed  the  task  for  2.5 
hours,  or  10  15-minute  blocks.  The  time  axis  measures  seconds  as  multiples  of  2-second  long  EEG 
epochs  which  were  not  all  contiguous,  due  to  rejection  of  EEG  segments  containing  movement  or  other 
artifacts.  Some  blocks  have  fewer  epochs  than  others  because  the  incidence  of  EEG  artifacts  increased 
during  those  blocks. 
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Figure  39.  Atomic  decomposition  of  EEG  from  participant  RWC  of  the  NASA-C  study.  EEG  recordings 
from  30  channels  were  processed  using  PARAFAC  decomposition  to  yield  a  model  consisting  of  four 
atoms,  each  have  dimensions  of  space  (electrodes),  frequency  (power  spectral  density)  and  time  (time  on 
task).  Graphical  conventions  are  the  same  as  in  Figure  32.  This  participant  performed  the  task  for  three 
hours,  or  12  15-minute  blocks.  The  time  axis  measures  seconds  as  multiples  of  2-second  long  EEG 
epochs  which  were  not  all  contiguous,  due  to  rejection  of  EEG  segments  containing  movement  or  other 
artifacts.  Some  blocks  have  fewer  epochs  than  others  because  the  incidence  of  EEG  artifacts  increased 
during  those  blocks. 
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Figure  40.  Atomic  decomposition  of  EEG  from  participant  RGR  of  the  NASA-C  study.  EEG  recordings 
from  30  channels  were  processed  using  PARAFAC  decomposition  to  yield  a  model  consisting  of  four 
atoms,  each  have  dimensions  of  space  (electrodes),  frequency  (power  spectral  density)  and  time  (time  on 
task).  Graphical  conventions  are  the  same  as  in  Figure  32.  This  participant  performed  the  task  for  2.5 
hours,  or  10  15-minute  blocks.  The  time  axis  measures  seconds  as  multiples  of  2-second  long  EEG 
epochs  which  were  not  all  contiguous,  due  to  rejection  of  EEG  segments  containing  movement  or  other 
artifacts.  Some  blocks  have  fewer  epochs  than  others  because  the  incidence  of  EEG  artifacts  increased 
during  those  blocks. 
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Figure  41.  Atomic  decomposition  of  EEG  from  participant  SKH  of  the  NASA-C  study.  EEG  recordings 
from  30  channels  were  processed  using  PARAFAC  decomposition  to  yield  a  model  consisting  of  four 
atoms,  each  have  dimensions  of  space  (electrodes),  frequency  (power  spectral  density)  and  time  (time  on 
task).  Graphical  conventions  are  the  same  as  in  Figure  32.  This  participant  performed  the  task  for  2.75 
hours,  or  1 1  15-minute  blocks.  The  time  axis  measures  seconds  as  multiples  of  2-second  long  EEG 
epochs  which  were  not  all  contiguous,  due  to  rejection  of  EEG  segments  containing  movement  or  other 
artifacts.  Some  blocks  have  fewer  epochs  than  others  because  the  incidence  of  EEG  artifacts  increased 
during  those  blocks. 
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Figure  42.  Atomic  decomposition  of  EEG  from  participant  TBN  of  the  NASA-C  study.  EEG  recordings 
from  30  channels  were  processed  using  PARAFAC  decomposition  to  yield  a  model  consisting  of  four 
atoms,  each  have  dimensions  of  space  (electrodes),  frequency  (power  spectral  density)  and  time  (time  on 
task).  Graphical  conventions  are  the  same  as  in  Figure  32.  This  participant  performed  the  task  for  1.5 
hours,  or  6  15-minute  blocks.  The  time  axis  measures  seconds  as  multiples  of  2-second  long  EEG  epochs 
which  were  not  all  contiguous,  due  to  rejection  of  EEG  segments  containing  movement  or  other  artifacts. 
Some  blocks  have  fewer  epochs  than  others  because  the  incidence  of  EEG  artifacts  increased  during  those 
blocks. 
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Figure  43.  Atomic  decomposition  of  WXS  from  participant  KTT  of  the  NASA-C  study.  EEG  recordings 
from  30  channels  were  processed  using  PARAFAC  decomposition  to  yield  a  model  consisting  of  four 
atoms,  each  have  dimensions  of  space  (electrodes),  frequency  (power  spectral  density)  and  time  (time  on 
task).  Graphical  conventions  are  the  same  as  in  Figure  32.  This  participant  performed  the  task  for  three 
hours,  or  12  15-minute  blocks..  The  time  axis  measures  seconds  as  multiples  of  2-second  long  EEG 
epochs  which  were  not  all  contiguous,  due  to  rejection  of  EEG  segments  containing  movement  or  other 
artifacts.  Some  blocks  have  fewer  epochs  than  others  because  the  incidence  of  EEG  artifacts  increased 
during  those  blocks. 
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Figures  for  Coherence  Analyses  of  12  Participants’  EEG  from  the  NASA-C  Database 


Figure  44  (next  page).  Atomic  decomposition  of  EEG  coherence  from  participant  ARB  of  the  NASA-C 
study.  EEG  coherences  from  30  channels  were  processed  using  PARAFAC  decomposition  to  yield  a 
model  consisting  of  four  atoms,  each  have  dimensions  of  space  (electrodes),  coherence  (magnitude 
squared  coherence)  and  time  (time  on  task).  Upper  left  panel:  distributions  of  loadings  for  Atoms  1-4 
(black,  blue,  green,  &  red  lines)  across  EEG  frequency  bins  (due  to  a  labeling  error  in  the  legend,  atoms 
are  numbered  0-3  in  this  panel,  but  we  shall  refer  to  them  everywhere  as  Atoms  1-4  in  the  same  order). 
Unlike  the  figures  for  the  APEX-W  and  APECS-F/PSD  analyses,  here  the  loadings  of  each  atom  across 
frequencies  were  normalized  to  the  range  of  0.0  to  1.0  for  all  atoms.  We  did  this  to  emphasize  the  shape 
of  the  coherence  loading  spectra,  as  the  raw  loadings  varied  considerably  in  scale  across  atoms.  Upper 
right  panel:  distributions  of  loadings  for  Atoms  1-4  across  time  on  task.  Unlike  the  PSD  analyses, 
where  all  blocks  were  analyzed  for  each  participant,  here  only  the  first  (alert)  30  minutes  and  last 
(fatigued)  30  minutes  of  time  on  task  were  analyzed  for  each  participant.  The  first  and  last  30  minutes 
are  separated  by  the  dashed  vertical  line.  Total  times  on  task  for  each  subject  and  the  description  of  the 
time  axis  appear  in  corresponding  figures  of  Appendix  3.  Lower  Four  Panels:  Modified  Werner 
projections  of  electrode  pairs  of  Atoms  1-4,  in  which  only  pairs  with  coherence  loadings  exceeding  the 
90th  percentile  of  all  coherence  loadings  for  that  atom  are  plotted.  Conventions  for  the  projection  are 
explained  in  Figure  3 1 .  Electrode  pairs  are  divided  into  three  groups,  based  on  linear  inter-electrode 
distances  on  straight  lines  through  the  sphere  (not  along  the  surface).  The  criteria  for  these  distances 
were  derived  from  a  spherical  head  model  with  a  radius  of  9  cm.  Short-range  pairs  are  connected  by 
wide  magenta  lines,  and  are  limited  to  pairs  whose  inter-electrode  distance  is  less  than  6.0  cm. 
Respectively,  medium-range  (6.0  to  12.0  cm)  pairs  and  long-range  (greater  than  12.0  cm)  pairs  are 
connected  by  medium-wide  cyan  lines  and  thin  orange  lines. 

Figures  45  to  55  (following  1 1  pages).  Atomic  decompositions  for  the  remaining  1 1  participants  of  the 
NASA-C  study.  The  figures  are  labeled  with  the  participants’  3 -letter  codes,  which  match  the  codes  of 
the  figures  in  Appendix  3.  Graphical  conventions  are  the  same  as  in  Figure  44. 


Page  A4-1 


APECS  Final  Report 
September  15,  2009 


ARO  Contract  No.  W91 1NF08CO121 
PDT  Report  No.  UA01BF0636B131 


Figure  44.  Coherence  analyses  for  participant  ARB.  Graphing  conventions  are  explained  in  Figure  44. 
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Figure  45.  Coherence  analyses  for  participant  GSD.  Graphing  conventions  are  explained  in  Figure  44. 
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Figure  46.  Coherence  analyses  for  participant  JCH.  Graphing  conventions  are  explained  in  Figure  44. 
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Figure  47.  Coherence  analyses  for  participant  JCS.  Graphing  conventions  are  explained  in  Figure  44. 
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KTT 


Figure  48.  Coherence  analyses  for  participant  KTT.  Graphing  conventions  are  explained  in  Figure  44. 
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Figure  49. Coherence  analyses  for  participant  MMB.  Graphing  conventions  are  explained  in  Figure  44. 
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Figure  50.  Coherence  analyses  for  participant  MMC.  Graphing  conventions  are  explained  in  Figure  44. 
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Figure  51.  Coherence  analyses  for  participant  RGR.  Graphing  conventions  are  explained  in  Figure  44. 
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Figure  52.  Coherence  analyses  for  participant  RWC.  Graphing  conventions  are  explained  in  Figure  44. 
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5KH 


Figure  53.  Coherence  analyses  for  participant  SKH.  Graphing  conventions  are  explained  in  Figure  44.  44. 
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TBN 


Figure  54.  Coherence  analyses  for  participant  TBN.  Graphing  conventions  are  explained  in  Figure  44. 
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Figure  55.  Coherence  analyses  for  participant  WXS.  Graphing  conventions  are  explained  in  Figure  44. 
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